k-dense-ai/claude-scientific-skills

Overview

This skill provides a compact, production-ready guide to using Polars—an ultra-fast, in-memory DataFrame library for Python and Rust built on Apache Arrow. It focuses on expression-based transformations, lazy evaluation, and high-performance patterns to replace pandas where data fits in RAM. Use it to speed up ETL pipelines, exploratory analysis, and scientific workflows on datasets roughly 1–100 GB.

How this skill works

The skill explains Polars primitives: Expressions (pl.col, pl.when, chaining), eager DataFrame vs LazyFrame with query planning, and I/O patterns (CSV, Parquet, JSON, cloud storage). It shows how Polars builds optimized plans for lazy queries, executes operations in parallel, and leverages Arrow memory layouts for low overhead. Examples cover selection, filtering, with_columns, group_by/agg, joins, window functions, and common migration mappings from pandas.

When to use it

• When pandas is too slow but data still fits in RAM (roughly 1–100 GB).
• For ETL pipelines that benefit from query optimization and parallel execution.
• When you need fast columnar I/O (Parquet) and efficient memory use via Apache Arrow.
• For migrating pandas code to a parallel, expression-based API.
• When you want lightweight lazy evaluation and predicate/projection pushdown.

Best practices

• Prefer LazyFrame (pl.scan_*) for large files to build and optimize query plans before collect().
• Keep computations inside the expression API—avoid Python loops or row-wise Python functions in hot paths.
• Select only required columns early to enable projection pushdown and reduce memory use.
• Use appropriate dtypes (categorical, correct integer sizes, date types) to save memory and speed up grouping.
• Use streaming or collect(streaming=True) for very large data and pl.scan_csv for partitioned files.

Example use cases

• Fast ETL: scan large CSVs lazily, apply filters/projections, and write optimized Parquet outputs.
• Scientific data analysis: group_by and window functions to compute per-sample statistics in genomics or proteomics.
• Pandas migration: convert pandas pipelines to pl.with_columns and pl.group_by for parallel speedups.
• Data reshaping: pivot/unpivot and joins for feature engineering in materials science or drug discovery.
• Ad-hoc analytics: interactive exploration on multi-GB datasets with low-latency queries.