data-quality_skill

Overview

This skill implements systematic data validation, quality metrics, and cleansing strategies to ensure accuracy, completeness, and consistency across datasets. It provides rule-based checks, automated monitoring, and corrective actions that integrate into ETL pipelines or migration workflows. The goal is measurable, repeatable improvement in data trustworthiness for analytics and downstream systems.

How this skill works

The skill defines quality dimensions (accuracy, completeness, consistency, timeliness, validity, uniqueness) and applies validation rules like type, range, format, referential integrity, and business constraints. It computes metrics (error rates, completeness percentages, duplicates, data age), collects issues with severity levels, and generates actionable reports. Cleansing routines remove duplicates, standardize formats, fill or flag missing values, and recalculate derived fields. Alerts and dashboards surface threshold breaches and trends.

When to use it

• Building or hardening data pipelines and ETL processes
• During data migration or system integration projects
• Implementing data governance and SLAs
• Setting up continuous data monitoring and alerting
• Investigating and resolving recurring data quality incidents

Best practices

• Validate data at both source and target systems before processing
• Define quality metrics and acceptable thresholds with business owners
• Automate checks in pipelines and log every fix or quarantine action
• Monitor trends over time and alert on threshold breaches immediately
• Document rules, rationale, and remediation steps for audits and handoffs

Example use cases

• Sales pipeline: validate required fields, prices, totals, and referential integrity with customer/product master tables
• Migration: verify completeness and uniqueness after a bulk transfer, flag missing foreign keys
• Operational monitoring: measure data age and alert when feeds fall behind SLA
• Cleansing job: remove exact duplicates, standardize dates, fill defaults and recalculate derived totals
• Reporting readiness: run a quality gate that blocks downstream reports until critical issues are resolved