ml-data-pipeline-architecture_skill

Overview

This skill captures proven architecture patterns for efficient ML data pipelines using Polars, Apache Arrow, and ClickHouse. It explains decision criteria, zero-copy data flows, lazy evaluation, and scalable PyTorch integration to reduce memory use and speed up ETL and training. The guidance is practical, migration-ready, and focused on production constraints.

How this skill works

I describe when to prefer Polars over Pandas and how to wire Arrow-backed transfers from ClickHouse into Polars for minimal copies. The patterns include Arrow stream ingestion, lazy scan APIs, Parquet batch flows, and a memory-efficient bridge into PyTorch DataLoaders. I also cover schema validation, benchmarks, migration steps, and anti-patterns to avoid common traps.

When to use it

• Choosing Polars vs Pandas based on dataset size and operation type
• Building zero-copy pipelines from ClickHouse into ML training
• Optimizing memory and throughput for >1M-row datasets
• Converting batch jobs to lazy, predicate-pushed pipelines
• Feeding large datasets into PyTorch without loading full dataset into RAM

Best practices

• Prefer Polars for >1M rows and heavy group-by or join workloads
• Use ClickHouse → ArrowStream → pl.from_arrow for zero-copy ingestion
• Compose transformations with pl.scan_* and call .collect() only at the end
• Validate schemas early and fail-fast with clear error messages
• Avoid round-tripping to pandas; keep data in Arrow/Polars to prevent extra copies

Example use cases

• Stream large ClickHouse result sets into a Polars-backed training pipeline with 1.2x peak memory
• Migrate a Pandas-based ETL to Polars using Arrow transfer and lazy scans to cut runtime by 5–10x
• Serve batches to PyTorch using a PolarsDataset that slices an Arrow table for per-row access
• Export reproducible batch inputs to Parquet, process lazily, and apply predicate pushdown
• Implement schema-driven feature validation with Pydantic before model training