debugging-dags_skill

Overview

This skill performs comprehensive DAG failure diagnosis and root cause analysis for Apache Airflow pipelines. It guides a systematic investigation from identifying the failing run to delivering immediate fixes and long-term prevention recommendations. Use it for complex, multi-faceted debugging that requires deep context and structured remediation.

How this skill works

The skill inspects DAG run and task state, fetches and parses task logs to extract the real exception, and classifies failures into data, code, infrastructure, or dependency issues. It compares failed runs to recent successful runs, checks recent code or data changes, and assembles a focused Root Cause, Impact Assessment, Immediate Fix, and Prevention plan. It also supplies ready-to-run CLI commands to validate fixes and rerun tasks.

When to use it

• When a pipeline failure needs full root cause analysis beyond simple log viewing
• When failures recur intermittently or across multiple DAGs
• When the cause is unclear and requires correlation of logs, run metadata, and upstream health
• When you must provide remediation steps and prevention guidance to engineers or SREs
• When a production report or downstream process is blocked and you need an actionable recovery plan

Best practices

• Start by identifying the specific failed DAG run before deep investigation
• Always extract the actual exception from task logs, ignoring Airflow boilerplate
• Classify the failure (data, code, infra, dependency) to focus remediation
• Compare the failed run to recent successful runs and recent code/data changes
• Provide precise, executable remediation: SQL fixes, code patches, or infra actions, plus commands to clear and rerun tasks
• Include prevention: data quality checks, retries/backoff, alerting, and documentation updates

Example use cases

• Diagnose a DAG that intermittently fails with downstream data inconsistencies
• Perform a full root cause analysis after a production report went stale
• Investigate repeated import errors after a deploy and recommend a rollback or patch
• Analyze resource exhaustion errors and produce an infra remediation and monitoring plan
• Produce step-by-step remediation commands and rerun instructions for on-call engineers