data-cleaning-pipeline_skill

Overview

This skill generates reproducible data cleaning pipelines for pandas, polars, or PySpark. It focuses on practical handling of missing values, duplicates, outliers, type conversions, text normalization, and data validation with logging and reporting.

How this skill works

Given a dataset, the skill analyzes structure and common issues, then produces a step-by-step pipeline implementing deduplication, missing-value strategies, type fixes, outlier filtering, categorical encoding, and range checks. Output includes runnable code snippets for pandas, polars, or PySpark and a cleaning report that documents all changes.

When to use it

• You need a ready-to-run cleaning pipeline for an uploaded CSV or DataFrame
• You want to handle missing values, choose fill strategies, or drop high-missing columns
• You need to remove duplicates and keep a reproducible log of what changed
• You must detect or remove outliers using IQR or z-score methods
• You want to normalize text fields and convert column types before modeling

Best practices

• Keep the original raw data and work on a copy; always log each cleaning step
• Drop columns only when a clear missing-data threshold is met (e.g., >50%)
• Use median for skewed numeric fills and mode or 'Unknown' for categorical fills
• Validate types and ranges after conversions; retain a validation report
• Prefer polars or PySpark for large datasets; use pandas for exploratory cleaning

Example use cases

• Generate a pandas pipeline that removes duplicates, fills missing ages with median, encodes categories, and outputs a cleaning report
• Create a polars script to process large CSVs: fast deduplication, null fills, date parsing, and text normalization
• Produce a PySpark pipeline for big-data workflows that casts columns, filters out-of-range values, and writes cleaned partitions
• Auto-generate a YAML configuration for a repeatable cleaning workflow deployed in a CI pipeline
• Run data quality checks to find duplicate IDs, invalid emails, and out-of-range ages before model training