data_skill

Overview

This skill teaches practical data engineering and analytics: building ETL/ELT pipelines, designing data warehouses, optimizing SQL, and preparing data for analytics or ML. It focuses on scalable tooling and real-world patterns so you can move from raw events to reliable, queryable datasets quickly. The guidance covers batch and streaming, testing, monitoring, and governance to keep pipelines production-ready.

How this skill works

The skill inspects pipeline design, transformation logic, and operational patterns to recommend improvements and templates. It evaluates data ingestion, staging, transformation, and serving layers using SQL snippets, orchestration patterns, and tooling choices. Recommendations include query optimizations, partitioning and clustering, schema design for analytics, and monitoring/test strategies.

When to use it

• Building or refactoring ETL/ELT pipelines for batch or streaming data
• Designing a data warehouse schema or choosing a warehousing platform
• Optimizing slow SQL queries and analytics performance
• Preparing datasets for machine learning or BI reporting
• Establishing data quality, testing, and monitoring practices

Best practices

• Implement data quality checks and automated validation at every stage
• Use version control for transformations and modular, testable code (dbt, CI)
• Design partitions and clustering based on query patterns and cardinality
• Instrument pipelines with alerts, metrics, and lineage for observability
• Apply least-privilege governance and encrypt sensitive data in transit and at rest

Example use cases

• Create daily staging tables from raw event streams and produce aggregated metrics for dashboards
• Migrate on-prem ETL jobs to a cloud data warehouse with optimized SQL and partitioning
• Build a streaming pipeline using Kafka + Spark/Flink to power real-time analytics
• Implement dbt-based transformations with automated tests and CI/CD for analytics models
• Tune slow BI queries by adding appropriate indexes, materialized views, or denormalized tables