data-pipeline-designer_skill

Overview

This skill designs robust data pipelines using modern data stack patterns for ELT/ETL workflows. It outputs architecture JSON, orchestration templates, quality checks, and an implementation guide tuned to batch, streaming, or hybrid needs. The focus is on production readiness: idempotent DAGs, dbt layering, Great Expectations suites, and observability.

How this skill works

Given validated inputs about sources, targets, transformations, quality rules, and orchestration preferences, the skill classifies the pipeline type and selects patterns accordingly. It generates a pipeline_architecture JSON, Airflow/ksqlDB/dbt templates, quality-check artifacts, and monitoring configuration. Abort and TODO conditions are emitted when inputs are incomplete or invalid to ensure safe, idempotent outputs.

When to use it

• Designing a new batch, streaming, or hybrid pipeline
• Migrating ETL to ELT and adopting dbt transformation patterns
• Implementing or hardening data quality checks and SLAs
• Troubleshooting recurring pipeline failures or data quality regressions
• Establishing lineage, governance, and monitoring for analytics workloads

Best practices

• Normalize time and validate inputs before design; emit TODOs for missing schemas or business logic
• Prefer Airflow DAGs with TaskGroups for batch/hybrid and Kafka/ksqldb for streaming
• Use dbt layering: staging → intermediate → marts; one model = one logical transform
• Implement Great Expectations checkpoints at ingress, post-transform, and pre-load; block on critical checks
• Design idempotent and atomic tasks, avoid large XCom payloads; use secret managers for credentials

Example use cases

• Daily ELT for e-commerce orders: incremental ingestion, dbt models for metrics, GE checks for uniqueness and positive totals
• Real-time clickstream: Kafka producers → partitioned topics → ksqlDB transforms → materialized views
• Hybrid reporting: streaming events fed to real-time dashboards plus nightly batch aggregates for compliance
• Production rollout: Airflow DAG skeleton, dbt project scaffold, Great Expectations suite, OpenLineage integration