data-quality_skill

Overview

This skill codifies data-quality best practices and ready-to-run patterns for validating and monitoring datasets in Python. It packages core principles, example checks, and pipeline integrations so teams can start enforcing quality controls within minutes. The content emphasizes automated validation, measurable SLAs, and actionable monitoring for production systems.

How this skill works

The skill inspects data across four core dimensions: completeness, accuracy, consistency, and timeliness. It provides reusable check functions, Great Expectations expectation suites, and pipeline hooks to validate batches or streams, raise exceptions on failures, and export metrics to observability systems. Results drive alerts, dashboards, and downstream gating in orchestration tools like Airflow.

When to use it

• Before promoting data to production tables or model training datasets
• In daily ETL/ELT jobs to fail fast on schema or business-rule violations
• For real-time stream validation to divert bad messages to a dead-letter queue
• When establishing data quality SLAs with upstream providers
• During onboarding of new data sources to confirm expected shape and ranges

Best practices

• Define measurable data quality rules and SLAs with domain owners
• Automate validations in the pipeline and fail fast on critical violations
• Use weighted data-quality scores combining completeness, accuracy, consistency, and timeliness
• Export validation metrics to time-series stores and surface them on dashboards
• Track lineage and quality metadata so issues can be traced back to sources

Example use cases

• Validate daily transaction files with a Great Expectations suite and block downstream processing on failures
• Run range, null, uniqueness and cross-field consistency checks in an Airflow task before loading to a warehouse
• Apply per-message validation on Kafka streams and route invalid messages to a dead-letter topic
• Calculate and persist a rolling data quality score to monitor drift and trigger alerts
• Integrate quality checks into ML training pipelines to prevent model degradation from bad inputs