data-cleaning_skill

Overview

This skill teaches practical data cleaning, preprocessing, and quality assurance techniques for analytics workflows. It focuses on preparing reliable datasets by addressing missing values, outliers, type issues, duplicates, and string normalization. The goal is to make data analysis and modeling safer and faster by ensuring data quality at the source.

How this skill works

The skill inspects datasets to detect common problems: missing or malformed values, inconsistent types, duplicate records, outliers, and encoding or string issues. It provides methods for recovery such as imputation, deletion, chunked processing for large files, type conversion strategies, and validation against expected schemas. It also suggests error handling steps for memory limits and encoding mismatches.

When to use it

• Before exploratory data analysis or modeling to ensure input quality
• When you receive messy or unknown-format datasets from external sources
• During ETL pipelines to enforce consistent downstream data
• When encountering frequent type conversion, encoding, or memory errors
• Prior to reporting or BI dashboards to avoid misleading summaries

Best practices

• Profile data first to quantify missingness, distributions, and duplicates
• Prefer reproducible, programmatic cleaning steps over manual edits
• Use sampling or chunking for very large datasets to avoid memory errors
• Validate cleaned data against a schema and log assumptions and imputation
• Treat string normalization and encoding early to prevent downstream bugs

Example use cases

• Imputing missing demographic fields before building a customer churn model
• Removing duplicate rows and normalizing categorical labels for reporting
• Detecting and capping outliers in transaction amounts to stabilize modeling
• Converting and validating date and numeric formats from CSV imports
• Chunked processing of multi-gigabyte logs to handle memory constraints