dataform-engineering-fundamentals_skill

Overview

This skill codifies Dataform engineering fundamentals for BigQuery workflows, enforcing test-driven development, safe execution patterns, and maintainable architecture. It guarantees consistent use of source declarations, ${ref()} references, .sqlx declarations, and mandatory column documentation to prevent technical debt and production incidents. Follow these rules even under time pressure.

How this skill works

The skill inspects SQLX files, source declarations, config blocks, and pipeline execution commands to ensure TDD-first workflow, proper use of ${ref()}, and required safety flags like --schema-suffix dev and --dry-run. It verifies that new source declarations are .sqlx, that tables include comprehensive columns:{} documentation, and that operations/tests do not set explicit schema. It also enforces recommended architecture patterns (layering, incremental vs full refresh, assertions).

When to use it

• When creating or modifying BigQuery Dataform SQLX transformations or source declarations
• Before running pipelines, especially under deadlines or in ad-hoc/quick-report scenarios
• When adding or migrating source declarations (prefer .sqlx for new ones)
• When troubleshooting failing pipelines or validating data quality
• When designing repository structure, naming conventions, or preventing technical debt

Best practices

• Always write tests first (RED), implement minimal SQLX to pass tests (GREEN), then refactor
• Never hardcode table paths; always reference tables with ${ref()} after declaring sources
• Use dataform run --schema-suffix dev and run --dry-run to test safely before executing
• Document every table with a comprehensive columns:{} block; copy descriptions from sources or business docs
• Keep operations and assertion files free of schema: config to avoid schema conflicts
• Prefer layered repository structure (sources → intermediate → output) and break complex logic into intermediate steps

Example use cases

• Add a new customer_metrics table: write assertions first, implement SQLX, run tests with --schema-suffix dev
• Migrate a legacy .js source declaration: convert to .sqlx with full columns:{} documentation when modifying
• Troubleshoot a failing pipeline: validate with --dry-run, confirm ref() usage and source declarations, then run assertions
• Create incremental tables for daily events with uniqueKey and partitioning config
• Audit repository for hardcoded table paths and missing column docs to prevent downstream breakage