torchtitan_skill

Overview

This skill provides PyTorch-native distributed LLM pretraining using TorchTitan with composable 4D parallelism (FSDP2, tensor parallelism, pipeline parallelism, context parallelism). It scales from single-node 8 GPUs up to 512+ GPUs and supports Float8, torch.compile, and robust distributed checkpointing for models like Llama 3.1 and DeepSeek V3. The skill is focused on production-grade pretraining performance and interoperability with Hugging Face formats.

How this skill works

The skill configures and launches large-scale training jobs via TOML job configs and torchrun/srun wrappers, orchestrating FSDP2, TP, PP and CP degrees across nodes and GPUs. It automates tokenizer and HF asset downloads, enables Float8 through torchao converters, and integrates torch.compile for faster kernels. Checkpointing and seed-checkpoint creation support consistent PP initialization and resuming across different parallelism layouts.

When to use it

• Pretraining LLMs from scratch (8B to 405B+).
• You need a PyTorch-native solution without heavy third-party runtime dependencies.
• When you require composable 4D parallelism (FSDP2, TP, PP, CP) for memory and throughput scaling.
• Training on H100s with Float8 and compile-based speedups.
• When interoperable checkpoints (Hugging Face / DCP) and auto-resume are required.

Best practices

• Start with a seed checkpoint before enabling pipeline parallelism to ensure consistent initialization across PP stages.
• Enable selective or full activation checkpointing and gradient accumulation to avoid OOM on large models.
• Use Float8 only for large GEMMs; filter small layers to prevent performance regressions.
• Test configurations on a small node (8 GPUs) before scaling to many nodes; monitor TensorBoard logs.
• Set TORCH_NCCL_AVOID_RECORD_STREAMS=1 to avoid TP async-collective memory spikes when necessary.

Example use cases

• Single-node pretraining of Llama 3.1 8B using an editable TOML config and torchrun for reproducible runs.
• Multi-node SLURM deployment for a 70B model across 32 nodes with explicit TP and FSDP sharding.
• Large-scale 4D training of a 405B model across 512 GPUs with seed checkpoint creation and distributed checkpointing.
• Enable Float8 + torch.compile on H100 clusters to get 30–50% throughput speedups for production pretraining.
• Convert sharded checkpoints to monolithic Torch checkpoints for downstream evaluation or HF conversion.