architecting-data_skill

Overview

This skill provides strategic guidance for designing modern data platforms, including storage paradigms (lake, warehouse, lakehouse), modeling approaches, medallion patterns, table-format selection, and governance frameworks. It helps architects and platform engineers choose patterns and tools that balance cost, performance, and organizational readiness for BI and ML workloads.

How this skill works

The skill inspects your primary use cases, budget, organizational scale, and data maturity to recommend storage paradigms and modeling patterns. It applies decision trees for storage, modeling, table-format selection, and data-mesh readiness, and maps those decisions to a recommended tool stack and implementation patterns (medallion layers, incremental loads, governance). It also highlights anti-patterns and concrete migration or startup recommendations.

When to use it

• Designing a new data platform or modernizing a legacy warehouse
• Choosing between data lake, warehouse, or lakehouse for BI and ML workloads
• Selecting a data modeling approach: dimensional, normalized, data vault, or wide tables
• Evaluating open table formats (Iceberg, Delta Lake, Hudi) for ACID on object storage
• Assessing whether to adopt a data mesh or keep a centralized platform
• Designing medallion layers (bronze/silver/gold) and governance/catalog requirements

Best practices

• Start simple: prefer a warehouse or basic lakehouse and iterate rather than over-engineering
• Adopt medallion architecture to separate raw, cleaned, and business-level data
• Invest in governance early: catalog, lineage, data quality tests, and access controls
• Prefer open table formats (Apache Iceberg recommended) to reduce vendor lock-in
• Automate data quality and CI/CD for transformations (dbt, Great Expectations)
• Assess data-mesh readiness with a multi-factor score; avoid decentralizing prematurely

Example use cases

• Startup (50 people) with transactional + product analytics: recommend simple cloud warehouse + Airbyte/Fivetran + dbt
• Enterprise migration from legacy Oracle: incremental move to lakehouse with medallion layers and CDC
• ML platform needing raw feature access: lakehouse with wide tables and Iceberg for multi-engine access
• Company debating data mesh: run readiness assessment; recommend hybrid or central-first if score is low
• Real-time CDC-heavy pipelines: object store with Apache Hudi or Iceberg and Kafka for ingestion