miles_skill

Overview

This skill provides enterprise-grade guidance for reinforcement learning training using miles, a production-ready fork of slime tailored for large MoE models and low-precision pipelines. It focuses on stable FP8/INT4 training, bit-wise train-inference alignment, and speculative RL for maximum rollout throughput. The content includes practical workflows, configuration examples, and troubleshooting tips for production deployments.

How this skill works

The skill explains how miles configures and runs large-scale RL training by combining unified FP8 pipelines, INT4 quantization-aware training, and Rollout Routing Replay (R3) to ensure identical expert routing between inference and training. It details speculative RL with EAGLE draft models and online MTP updates to increase throughput, plus kernel-level optimizations (FlashAttention-3, DeepGEMM) and zero-copy weight sync for performance. The guidance includes commands, environment variables, and verification checks to validate stable runs.

When to use it

• Training very large MoE models (1TB+), e.g., DeepSeek V3 or Qwen3-MoE
• You need FP8 or INT4 quantization-aware training for production memory savings
• Require bit-wise identical train-inference alignment for MoE routing reproducibility
• Maximizing rollout throughput via speculative RL and partial rollouts
• Deploying stable, enterprise-grade RL with kernel and IPC optimizations

Best practices

• Enable FP8 block scaling and set CUDA connection limits for FP8 stability
• Verify routing consistency and check for NaN/Inf immediately after model load
• Use R3 to record and replay expert routing for exact train-inference alignment
• Start with conservative speculative steps and enable online MTP if draft acceptance drifts
• Use Docker images for reproducible environments and pin GPU drivers/ CUDA versions

Example use cases

• Train Qwen3-MoE with INT4 QAT to fit 1TB-class models on H200 hardware
• Run speculative RL with EAGLE to gain 25–40% rollout throughput and add partial rollout for extra gains
• Deploy R3 when MoE routing mismatch causes RL instability or reward collapse
• Use unified FP8 to avoid quantization-induced RL collapse during large-scale MoE fine-tuning