transforming-data_skill

Overview

This skill transforms raw data into production-ready analytical assets using ETL/ELT patterns, dbt, Python dataframes (pandas, polars, PySpark), and orchestration tools like Airflow. It helps design incremental models, migrate pandas workloads to polars, and build tested, documented pipelines. The goal is reliable, performant transformations with built-in testing and monitoring.

How this skill works

The skill recommends a pattern (ETL, ELT, or hybrid) based on infrastructure, compliance, and data volume. It provides concrete implementation templates: dbt models and tests, polars/pandas/PySpark transformation snippets, and Airflow DAG patterns. It also includes guidance on incremental loading, idempotency, error handling, and data quality checks using dbt tests or Great Expectations.

When to use it

• Choosing ETL vs ELT for a new pipeline
• Building dbt models (staging → intermediate → marts)
• Implementing incremental loads and merge strategies
• Migrating pandas code to polars for speed and memory efficiency
• Orchestrating multi-step pipelines with Airflow, Dagster, or Prefect
• Adding automated data quality tests and monitoring

Best practices

• Default to ELT with dbt for analytics unless compliance or target constraints force ETL
• Structure dbt into staging, intermediate, and marts; prefer incremental materializations for large facts
• Prefer polars for 500MB–100GB workloads; use pandas for small prototypes and PySpark for >100GB distributed jobs
• Make incremental models idempotent using merge/unique_key and created_at filters
• Add automated tests (dbt generic/singular tests or Great Expectations) and surface failures via alerts
• Instrument pipelines with row counts, SLAs, and failure notifications in Airflow or chosen orchestrator

Example use cases

• Daily sales pipeline: extract raw events, load to warehouse, dbt staging plus incremental fact model, monitored by Airflow
• Pandas-to-polars migration: convert eager pandas scripts to polars scan/with_columns/collect for large CSVs
• Large-scale aggregation: use PySpark on cluster for >100GB joins and window calculations
• Hybrid flow: pre-load PII redaction in ETL then ELT analytics in warehouse with dbt
• Data quality ramp-up: add dbt tests and Great Expectations suites and fail pipelines on violations