annotating-task-lineage_skill

Overview

This skill annotates Airflow tasks with data lineage by setting inlets and outlets on operators. It provides patterns and examples for using OpenLineage Dataset objects, Airflow Assets, or Airflow Datasets to declare inputs and outputs. Use it to add lightweight, table- or file-level lineage where operators lack built-in lineage extractors. It is designed for quick setup and clear dataset naming across platforms.

How this skill works

The skill attaches OpenLineage Dataset or Airflow Asset/Dataset objects to operators via the inlets and outlets parameters, or by setting these attributes dynamically in operator execute() methods. OpenLineage will use inlets/outlets as a fallback if no extractor or operator-level OpenLineage methods are present. Naming helper classes are recommended to construct consistent namespace and name values for storage systems and warehouses.

When to use it

• Operator has no OpenLineage extractor or get_openlineage_* methods
• You need quick table- or file-level lineage without custom extractors
• You want to document task inputs and outputs at DAG authoring time
• You need to annotate multiple input sources or multiple outputs
• You need a fallback lineage mechanism when hook-level or extractors are not available

Best practices

• Always use OpenLineage naming helpers for consistent namespace and name values
• Document lineage intent in task comments and keep inlets/outlets in sync with SQL or code changes
• Include all tables referenced in JOINs as inlets and all written targets as outlets
• Prefer implementing get_openlineage_facets_on_* in custom operators for complex extraction
• Set inlets/outlets dynamically in execute() only for non-deferrable operators

Example use cases

• Annotate a BashOperator that reads from a Snowflake table and writes to S3 using Dataset objects
• Declare multiple source tables for an aggregate PythonOperator with several outputs
• Add lineage to a custom operator by setting self.inlets/self.outlets in execute() for simple cases
• Use naming helpers to build BigQuery, Snowflake, Postgres or S3 dataset identifiers
• Provide outlet-only annotations for tasks that only publish datasets used by downstream DAGs