data-engineering_skill

Overview

This skill is a data-engineering expert focused on designing scalable data pipelines, ETL/ELT processes, data warehousing, and analytics infrastructure. It provides concrete architecture patterns, technology recommendations, and implementation examples across batch, streaming, and ELT workflows. Implementations and examples target production-grade systems and include Rust-native orchestration and integration approaches.

How this skill works

It inspects pipeline design choices and recommends architecture patterns (batch, streaming, ELT) based on source types, latency needs, and scale. It evaluates component selections—ingestion, processing, storage, transformation, and orchestration—and maps them to concrete technologies and deployment patterns. It also provides schema and modeling guidance for dimensional and normalized warehouses, plus code and config examples to bootstrap implementations.

When to use it

• Designing a new analytics platform or migrating to a modern data stack
• Choosing between batch, streaming, or ELT based on latency and cost requirements
• Selecting ingestion and processing technologies for high-throughput sources
• Creating data warehouse schemas and partitioning strategies for large datasets
• Implementing production ETL/ELT jobs with monitoring, retries, and idempotency

Best practices

• Start with clear SLAs for freshness and latency and pick patterns that meet them
• Use immutable raw layers in the data lake or warehouse and centralize transformations in dbt or equivalent
• Partition and cluster large fact tables and use surrogate keys for dimensional joins
• Test transformations with unit and integration tests and validate schema evolution
• Design idempotent, resumable pipelines and implement retries, backfill paths, and observability

Example use cases

• Batch ETL: nightly Spark job joining OLTP and S3 sources, writing Parquet to a data lake and summarized tables to Snowflake
• Streaming ETL: Kafka → Flink job for event enrichment and routing to processed topics and low-latency stores
• ELT with dbt: ingest SaaS APIs into BigQuery/Snowflake, then transform with dbt models into marts for BI
• Warehouse design: star schema for e‑commerce orders with partitioned fact tables and lookup dimensions
• Orchestration: Airflow or Dagster DAGs for complex dependencies, retries, and SLA alerts