pytorch-fsdp_skill

Overview

This skill provides expert guidance for training large models with PyTorch Fully Sharded Data Parallel (FSDP), covering parameter sharding, mixed precision, CPU offload, and the FSDP2 workflow. It distills core APIs, common patterns, and operational advice so you can design efficient, scalable training pipelines. The content is focused on practical decisions and debugging tips for production and research use.

How this skill works

The skill inspects FSDP configuration choices (sharding strategies, mixed precision, activation and parameter offloading) and explains their runtime effects on memory, communication, and compute. It maps common APIs—like init_process_group, FSDP wrappers, and torch.distributed primitives—to concrete patterns such as uneven-input handling with Join and backend selection for NCCL/Gloo. It also highlights trade-offs when enabling CPU offload, gradient checkpointing, or FSDP2-style state management.

When to use it

• You need to train very large models that exceed single-GPU memory.
• You want to reduce GPU memory by sharding optimizer state and parameters.
• You plan to use mixed precision and need guidance on stability vs. performance.
• You encounter hangs or size mismatches in collective operations and need debugging patterns.
• You must choose or tune a distributed backend (NCCL, Gloo, MPI) or NIC interface.

Best practices

• Prefer NCCL for multi-GPU CUDA training; use Gloo for CPU-only setups and MPI when required by network hardware.
• Start with per-parameter or full-parameter sharding policies that match your optimizer pattern, then profile memory/comm overhead.
• Use mixed precision (AMP) for speed, but maintain master weights or dynamic loss scaling to avoid underflow.
• Enable CPU offload only after measuring GPU memory pressure—offloading reduces peak GPU memory but increases host-GPU transfers.
• Use Join context manager patterns for uneven-input workloads to avoid hangs and inconsistent collectives.
• Set NCCL and Gloo socket env vars (NCCL_SOCKET_IFNAME, GLOO_SOCKET_IFNAME) on multi-interface hosts and enable NCCL debug logging when diagnosing hangs.

Example use cases

• Sharding a transformer model’s parameters and optimizer state across GPUs to fit a 100B-parameter model.
• Combining FSDP with mixed precision to double throughput while keeping training stable.
• Using CPU offload for optimizer state when GPU memory is limited on inference-heavy research rigs.
• Debugging a hang in distributed training by enabling NCCL_DEBUG_SUBSYS and validating collective sizes.
• Implementing uneven-batch training where different ranks process different numbers of samples using Join and JoinHook.