apache-airflow-orchestration_skill

Overview

This skill is a complete guide to Apache Airflow orchestration that covers DAG design, operators, sensors, XComs, task dependencies, dynamic workflows, and production deployment. It focuses on practical patterns and production-ready choices so you can build, test, and operate reliable data pipelines. The content is hands-on and organized around real orchestration needs.

How this skill works

The skill inspects and explains core Airflow concepts: DAG definitions, operator types (Bash, Python, Email, Empty, custom), sensors, executors, and the scheduler. It describes dependency patterns (chains, fan-in/fan-out, branching), XCom usage for task communication, and deployment patterns for executors like Celery and Kubernetes. Examples and snippets illustrate common workflows, task grouping, edge labeling, and sensor strategies for event-driven pipelines.

When to use it

• Building and scheduling ETL/ELT pipelines with complex dependencies
• Coordinating multi-step data transformations across services
• Implementing event-driven or asset-based workflows that wait on external signals
• Creating dynamic DAGs that generate tasks programmatically
• Scaling execution across clusters using Celery or Kubernetes executors

Best practices

• Define DAGs programmatically in Python and keep logic testable and modular
• Prefer deferrable sensors or reschedule mode for long waits to free worker slots
• Use TaskGroups and clear task_ids for readability and easier troubleshooting
• Keep side-effectful code out of DAG parse time; perform work inside operators
• Propagate small, structured XCom payloads and avoid large binary transfers

Example use cases

• Daily ETL pipeline: extract -> transform -> load with XCom-based handoff
• Event-driven load: FileSensor waits for file, then triggers processing tasks
• Cross-DAG dependency: ExternalTaskSensor coordinating upstream DAG runs
• Dynamic task generation: create thousands of per-file tasks with chain()
• Production deployment: KubernetesExecutor for isolated, scalable task pods