moe-training_skill

Overview

This skill trains Mixture of Experts (MoE) models using DeepSpeed or HuggingFace tooling to scale model capacity while reducing compute cost. It targets sparse architectures like Mixtral 8x7B and DeepSeek-V3, enabling large parameter capacity with only a fraction of active compute. The package includes example architectures, DeepSpeed configs, routing and load-balancing losses, and inference optimizations for sparse expert activation.

How this skill works

The skill provides building blocks: expert FFNs, router/gating networks, top-k routing (e.g., top-1 or top-2), and auxiliary losses for load balancing and router stability. It supplies DeepSpeed and HuggingFace integration patterns, expert-parallel config snippets, and training scripts that handle capacity factors, token dropping, and expert sharding. For inference it includes sparse loading strategies to activate only the top-k experts per token and reduce memory and latency.

When to use it

• When you need to scale model capacity beyond dense compute limits (e.g., 47B capacity with 13B active compute).
• When training large models on limited hardware to reduce cost (typical 3–5× cost reduction vs dense).
• When specializing sub-networks (experts) for domains, languages, or tasks.
• When building or replicating state-of-the-art MoE models like Mixtral, Switch Transformer, or DeepSeek-V3.
• When you need lower inference latency and memory by sparsely activating experts.

Best practices

• Choose expert counts to match model size and data diversity (e.g., 8–16 for 1–7B, 8–64 for 7–30B, 64+ for 30B+).
• Use lower learning rates and longer decay schedules than dense models (start ~1e-4 for MoE).
• Tune capacity factors: smaller for training (1.25) to speed up, larger for evaluation (2.0) to avoid drops.
• Add auxiliary load-balancing loss and a router z-loss to stabilize routing and prevent expert collapse.
• Shard experts with expert_parallel_size in DeepSpeed and enable FP16/Zero optimizations for memory efficiency.

Example use cases

• Training Mixtral-style top-2 MoE with 8 local experts per layer to get large capacity at lower compute.
• Implementing PR-MoE (pyramid-residual) to improve parameter efficiency across layers.
• Using expert-choice routing for perfect load balancing in high-throughput production training.
• Offloading and sparse-loading experts at inference to serve large-capacity models with limited memory.
• Adapting capacity and expert counts to prevent overfitting on smaller datasets.