senior-data-engineer_skill

Overview

This skill delivers senior-level data engineering expertise for designing and operating production-grade data pipelines, ETL/ELT systems, and data infrastructure. It combines practical knowledge of Python, SQL, Spark, Airflow, dbt, Kafka and cloud platforms with patterns for scalability, reliability, and cost efficiency. Use it to architect systems, improve pipeline performance, and establish DataOps and governance practices.

How this skill works

The skill inspects architecture, pipeline code, orchestration, and monitoring to identify bottlenecks, failure modes, and cost drivers. It recommends design patterns for batch and real-time processing, data modeling, and feature stores, and it prescribes concrete fixes: refactoring Spark jobs, adding Airflow best practices, implementing dbt models, or tuning Kafka throughput. It also outlines observability, testing, security, and deployment steps needed to move from prototype to production.

When to use it

• Designing or reviewing scalable data platform architectures
• Building or refactoring ETL/ELT pipelines for reliability and performance
• Implementing streaming or real-time inference systems
• Establishing DataOps, testing, and monitoring practices
• Optimizing cloud costs, throughput, or latency for data workloads

Best practices

• Adopt test-driven development and version control for all pipeline code
• Use modular data models and dbt for reproducible transformations
• Orchestrate with Airflow or equivalent, with retries, SLA alerts, and idempotency
• Instrument pipelines with metrics, tracing, and structured logs for observability
• Enforce security: encryption, access control, PII handling, and regular audits
• Automate deployments with CI/CD and use canary/feature-flag releases

Example use cases

• Refactor legacy ETL to Spark for horizontal scaling and cost reduction
• Build a Kafka-backed real-time feature pipeline for low-latency ML inference
• Design a golden-table architecture with dbt and incremental models
• Implement DataOps workflows: CI tests, deployment pipelines, and monitoring
• Tune Airflow DAGs and resource configs to meet P95 latency targets