masterkram/data-quality-skill

Overview

This skill diagnoses and fixes data quality problems across the full lifecycle: define, detect, clean, and measure. It provides profiling, glitch detection (missing values, duplicates, outliers, constraint violations), repair routines (imputation, deduplication, capping), and verification tools to quantify improvement. Use it to prepare datasets for analysis or machine learning and to enforce data constraints before downstream use.

How this skill works

First, the skill profiles a dataframe to generate summary statistics and a quality report that highlights common glitches. It runs targeted detectors for missingness, duplicate/near-duplicate records, anomalies, and constraint violations (uniqueness, not-null, functional dependencies, domain limits). Cleaning functions apply context-appropriate repairs: listwise deletion or imputation for missing data, clustering and merge strategies for duplicates, capping or removal for outliers, and domain/constraint fixes. Finally, it re-profiles and validates constraints to measure improvement and produce a before/after summary.

When to use it

• Preparing raw data for exploratory analysis or model training
• Investigating unexplained model errors or data drift
• Removing duplicates and resolving entity records (entity resolution)
• Imputing or assessing missing data mechanisms before imputation
• Detecting and handling outliers that distort statistics or models
• Validating business rules and referential integrity across tables

Best practices

• Start with profiling to prioritize the most impactful issues (completeness, uniqueness, domain violations).
• Test missingness mechanisms (MCAR/MAR/MNAR) before choosing deletion vs imputation.
• Use blocking and similarity thresholds when matching to keep deduplication scalable.
• Prefer transparent, reproducible repair strategies (document rules, keep originals).
• Validate fixes by re-running profiles and constraint checks to quantify improvement.

Example use cases

• Profile a customer table, detect duplicate accounts by name/email, and merge using a most-complete-field strategy.
• Analyze missing income values, run MCAR tests, and apply regression imputation using correlated predictors.
• Detect price outliers with IQR rules, cap extreme values, and compare distribution before/after cleaning.
• Validate functional dependencies and domain constraints to ensure downstream data integrity for reporting.
• Cluster fuzzy matches for supplier names using Jaro-Winkler, then deduplicate and preserve audit trails.