deepspeed_skill

Overview

This skill provides expert guidance for distributed training with DeepSpeed, covering ZeRO optimization stages, pipeline and tensor parallelism, mixed-precision modes (FP16/BF16/FP8), 1-bit Adam, and sparse attention techniques. It focuses on practical configuration, performance tuning, and integration patterns to scale large models efficiently. Use it to design, debug, and optimize multi-node training pipelines and I/O-intensive workflows.

How this skill works

The skill inspects model and cluster topology, recommends ZeRO stage and memory-offload strategies, and explains parallelism trade-offs (data, tensor, pipeline). It provides concrete guidance on precision choices, optimizer offloading (including 1-bit Adam), and DeepNVMe/GDS I/O patterns for fast parameter swap and checkpointing. It also outlines performance tuning steps and common debugging checks.

When to use it

• Setting up ZeRO (stage 1/2/3) for memory and compute trade-offs
• Choosing FP16/BF16/FP8 and handling numerical stability
• Designing pipeline or tensor parallelism across GPUs/nodes
• Implementing low-communication optimizers like 1-bit Adam
• Tuning DeepNVMe/GDS I/O for checkpointing and parameter swap
• Troubleshooting training instability, OOMs, or poor scaling

Best practices

• Start with the simplest ZeRO stage that meets memory needs; only upgrade to higher stages when necessary to reduce comms cost
• Prefer BF16/FP16 for throughput; test FP8 cautiously and validate gradients and loss scaling
• Use gradient accumulation to amortize communication; combine with ZeRO to reduce peak memory
• Tune DeepNVMe handle parameters (block_size, queue_depth, intra_op_parallelism) or run ds_nvme_tune for storage-heavy workloads
• Enable pinned tensors and non-blocking async_pwrite/async_pread with proper wait() synchronization to avoid data races
• Profile end-to-end (GPU util, NCCL, PCIe, NVMe) before micro-optimizations; prioritize network and I/O bottlenecks

Example use cases

• Training a 100B-parameter transformer with ZeRO-Infinity and DeepNVMe parameter swap
• Switching a model from FP16 to BF16 to resolve diverging loss while preserving speed
• Reducing cross-node bandwidth with 1-bit Adam during multi-node fine-tuning
• Parallelizing a multi-stage model with pipeline parallelism for longer context lengths
• Automating DeepNVMe tuning for faster checkpoint save/load in inference/serving pipelines