airflow_skill

Overview

This skill manages Apache Airflow operations for listing, running, testing, and debugging DAGs, inspecting tasks and logs, and checking system health and configuration. It provides a consistent CLI surface for common Airflow workflows and quick troubleshooting, with instance management and direct REST API access when needed.

How this skill works

Commands run via a compact CLI wrapper that targets one or more Airflow instances and returns structured JSON. The skill can discover and persist instance configurations, list and inspect DAGs and runs, fetch task logs, trigger and wait for runs, diagnose failures, and call arbitrary Airflow REST endpoints for advanced use cases.

When to use it

• You need to list or inspect DAGs, their schedules, or source code.
• You want to trigger a DAG run, monitor progress, or run a blocking trigger-wait.
• A DAG run failed and you need diagnosis, task logs, or root-cause clues.
• You need to check Airflow health, version, pools, connections, or variables.
• You want to run ad-hoc REST API calls (XComs, event-logs, backfills) not covered by high-level commands.

Best practices

• Always run instance discovery in --dry-run first and obtain user consent before creating tokens.
• Store tokens in environment variables or the config file using ${VAR} references rather than committing secrets.
• Use af runs diagnose to gather context before pulling task logs for more efficient triage.
• Prefer af runs trigger-wait for reproducible, blocking execution during debugging; use trigger for asynchronous runs.
• Pipe JSON output to jq for targeted filtering and automation in scripts.

Example use cases

• Morning health check: af health → af dags errors → af config pools to spot broken DAGs and resource issues.
• Investigate a failure: af runs list --dag-id my_dag → af runs diagnose my_dag <run_id> → af tasks logs my_dag <run_id> <task_id>.
• Explore a DAG before changes: af dags explore my_dag to review metadata, tasks, and source code.
• Trigger and monitor a pipeline: af runs trigger-wait my_dag --timeout 1800 for end-to-end testing.
• Create or inspect runtime config: af config connections / af config variables and af api variables -X POST to manage via API.