domain-ml_skill

Overview

This skill provides practical guidance for building ML/AI applications in Rust with a focus on memory, GPU utilization, model portability, and high-throughput inference. It summarizes recommended crates, design patterns, and concrete code patterns for singleton model loading, batched inference, and streaming pipelines. Use it to make Rust ML systems efficient, reproducible, and production-ready.

How this skill works

The skill inspects domain constraints (large data, GPU, portability, precision, reproducibility) and maps them to Rust design implications and patterns. It recommends specific crates (tract, candle, tch-rs, burn, ndarray, polars) and shows how to implement singleton model loading, batch processing, and async data pipelines. It also enumerates common mistakes and fixes to avoid memory waste and GPU underutilization.

When to use it

• Building an inference service that loads ONNX models in Rust
• Training or running models on GPU with pure-Rust frameworks (candle, burn)
• Creating high-throughput batch inference pipelines
• Processing large datasets with zero-copy or streaming approaches
• Porting Python-trained models to Rust for production deployment

Best practices

• Avoid copying large tensors: use references, views, and in-place operations
• Load models once and reuse: use OnceLock or lazy initialization to prevent per-request loads
• Batch GPU work: collect inputs into batches to maximize GPU throughput
• Use standard model formats (ONNX) for portability between ecosystems
• Stream large data and use iterator-based pipelines or polars for lazy evaluation
• Seed RNGs and pin dependency versions for deterministic behavior

Example use cases

• Lightweight ONNX inference server using tract and OnceLock for model reuse
• Batch prediction pipeline: collect records, stack into batch tensors, run model, then unstack outputs
• GPU training/inference using candle or burn with async data loaders to hide IO latency
• Deploying PyTorch-trained models via tch-rs for direct interoperability with existing PyTorch artifacts
• Large-scale feature processing with polars and ndarray views to avoid copies