ml-failfast-validation_skill

Overview

This skill provides fail-fast POC validation patterns to catch common ML experiment issues before committing expensive compute. It bundles quick model, gradient, schema, prediction sanity, and NDJSON logging checks so experiments either start clean or fail fast with actionable messages. The patterns are implementation-agnostic and focused on minimizing wasted GPU/time during development.

How this skill works

The skill runs a short preflight suite that instantiates a model, verifies forward/backward passes, inspects gradient health, and performs mini-training and prediction sanity checks. It validates data and event schemas, NDJSON logging structure, checkpoint save/load, and timing budgets. When any check fails it raises clear, actionable errors so you fix the root cause before launching long experiments.

When to use it

• Before running a long training job or hyperparameter sweep
• When validating a new model architecture or data pipeline
• To check gradient flow and detect NaNs/Inf or zero gradients
• When adding new feature computation or data sources
• Before committing experiments to shared GPU resources

Best practices

• Run the full POC validation on CI or as a pre-commit/preflight step
• Prefer relative thresholds (e.g., pred_std / target_std) over absolute numbers
• Fail loudly: raise detailed errors rather than silencing exceptions
• Keep POC checks short (<2 minutes) and focused on actionable failures
• Log NDJSON events with required fields so downstream tooling can validate runs

Example use cases

• Quickly validate a new LSTM/transformer architecture for gradient flow and gate biases
• Checkpoint sanity: save then reload model weights before full training
• Detect collapsed or exploded predictions on a small test split to avoid wasted training
• Validate that feature pipeline produces required columns at pipeline boundaries
• Preflight NDJSON logging schema so experiment dashboards ingest clean events