deploying-airflow_skill

This skill helps you deploy Airflow DAGs and projects to production using Astro or Docker/Kubernetes CI/CD, streamlining deployment strategies.

Python

251

GitHub Stars

1

Bundled Files

2 months ago

Catalog Refreshed

3 months ago

First Indexed

Readme & install

Copy the install command, review bundled files from the catalogue, and read any extended description pulled from the listing source.

Installation

Preview and clipboard use veilstrat where the catalogue uses aiagentskills.

npx veilstrat add skill astronomer/agents --skill deploying-airflow

SKILL.md10.8 KB

Overview

This skill helps deploy Airflow DAGs and complete Airflow projects to staging and production. It covers managed deployments with Astro and self-hosted deployments using Docker Compose for development or Kubernetes with the official Helm chart for production. The guidance focuses on practical commands, CI/CD patterns, and deployment strategies to minimize downtime and speed iteration.

How this skill works

The skill inspects your deployment context (Astro vs open-source) and recommends appropriate commands and pipelines. For Astro it shows CLI deployment modes (full, DAG-only, image-only, dbt) and branch-to-environment mapping. For self-hosted setups it outlines Docker Compose quick starts for local/dev and Helm values and git-sync patterns for Kubernetes production deployments. It also highlights common operational commands and upgrade steps.

When to use it

You need to push DAG changes quickly without rebuilding images (fast iteration).
You want a managed CI/CD path with branch-to-environment mapping (Astro).
You're setting up a local development environment with Docker Compose.
You need a production-grade Airflow on Kubernetes using the Helm chart.
You want to design CI jobs that separate image builds from DAG pushes.

Best practices

Use astro deploy --dags for DAG-only changes and astro deploy for dependency or image changes.
Map branches to environments (main->prod, develop->staging) and automate deploys from CI.
For Kubernetes production, prefer git-sync or PV mounts over baking DAGs into images.
Pin Airflow image versions for reproducible releases and use Helm values.yaml for config.
Keep requirements changes in CI image builds and avoid ad-hoc pip installs on running clusters.

Example use cases

Fast feature iteration: run astro deploy --dags on a feature branch to validate DAG logic quickly.
Production release: CI merges to main trigger astro deploy (full) or Helm upgrade with a pinned image tag.
Local testing: bring up an Airflow 3 stack with docker compose up -d for development and debugging.
Kubernetes deploy: install or upgrade Airflow via helm install/upgrade and use git-sync for DAG distribution.
Multi-repo CI: build and push an image from one pipeline (astro deploy --image) and push DAGs from another (astro deploy --dags).

FAQ

Rebuild when you change dependencies, requirements.txt, plugins, the Dockerfile, or any non-DAG code that must be inside the image.

Which DAG deployment method is recommended for production on Kubernetes?

Use git-sync for automatic, reliable DAG sync from a repository. Persistent volumes or baked images are alternatives but git-sync is typically easiest to manage.