accelerate_skill

Overview

This skill provides a minimal, unified API to add distributed training to any PyTorch script in four lines. It standardizes device placement, mixed precision (FP16/BF16/FP8), and integration with DDP, DeepSpeed, FSDP, and Megatron while exposing an interactive configuration and a single launch command. The result is a consistent workflow across single-GPU, multi-GPU, multi-node, and TPU setups.

How this skill works

Accelerate wraps your model, optimizer, and dataloader with an Accelerator instance and handles device placement, mixed-precision autocasting, gradient scaling, and backward calls. It plugs into multiple backends (native DDP, DeepSpeed ZeRO, FSDP, Megatron) via plugins or config files and exposes helpers for accumulation, checkpointing, and process coordination. Launching is done with a single CLI command that reads the interactive config or a JSON backend config.

When to use it

• When you want to convert a single-GPU script to multi-GPU/multi-node with minimal code changes.
• When you need unified support for DeepSpeed, FSDP, DDP, or Megatron without switching APIs.
• When you want automatic device placement and mixed-precision support (FP16/BF16/FP8).
• When you prefer an interactive configuration and a single launch command across environments.
• When quick prototyping and compatibility with the HuggingFace ecosystem are priorities.

Best practices

• Do not manually .to('cuda') tensors or models after calling prepare(); let Accelerate handle placement.
• Use accelerator.backward(loss) and accelerator.accumulate(model) for correct gradient handling and accumulation.
• Run accelerate config to generate backend settings and select mixed precision or plugins before launching.
• Save and load checkpoints via accelerator.save_state and accelerator.load_state, and restrict saves to accelerator.is_main_process.
• Set seed with accelerate.utils.set_seed for reproducible results across sharded or FSDP runs.

Example use cases

• Convert a local single-GPU training script to an 8-GPU run by adding Accelerator.prepare and using accelerate launch.
• Enable BF16 on supported hardware for faster training with minimal code changes via mixed_precision='bf16'.
• Deploy ZeRO-2 DeepSpeed by passing a deepspeed config or deepspeed_plugin to Accelerator for large-model optimization.
• Use FSDP for ZeRO-3 equivalence and full-parameter sharding with an FSDP plugin and automatic wrapping.
• Apply gradient accumulation to simulate larger effective batch sizes across multiple devices with gradient_accumulation_steps.