debug-pytorch_skill

Overview

This skill helps you debug PyTorch issues systematically across training and inference. It targets common failures like CUDA OOM, shape mismatches, NaNs in loss/gradients, device or dtype conflicts, DataLoader worker problems, and autograd graph errors. It gives actionable checks, fixes, and tools to reproduce, isolate, and resolve problems quickly.

How this skill works

The skill walks you through four debugging phases: reproduce and isolate, validate data pipeline, validate model architecture, and inspect training dynamics. It provides concrete code snippets and diagnostics for memory profiling, anomaly detection, gradient inspection, device/dtype consistency, and DataLoader debugging. It also recommends targeted fixes such as gradient clipping, mixed precision, checkpointing, and tensor/device/dtype hygiene.

When to use it

• When you hit CUDA out of memory or unexplained GPU memory growth
• When encountering shape/size mismatch errors between layers or tensors
• When loss or gradients become NaN or infinite during training
• When runtime errors complain about CPU/GPU or dtype mismatches
• When DataLoader workers crash or you see broken pipe/fork errors
• When autograd errors mention in-place ops, missing grad_fn, or retain_graph issues

Best practices

• Reproduce failures with a minimal deterministic example and fixed random seeds
• Print or assert tensor shapes and devices at key model stages to catch mismatches early
• Use torch.autograd.set_detect_anomaly and check gradients after backward to locate NaNs/infs
• Prefer stable numerics: add eps where needed, use logits for CrossEntropyLoss, and clip gradients
• Manage GPU memory: reduce batch size, empty cache, use AMP/mixed precision and checkpointing
• Run DataLoader with num_workers=0 when debugging and set proper start method for CUDA in multiprocessing

Example use cases

• Resolve a CUDA OOM by reducing batch size, enabling torch.cuda.empty_cache, and switching to autocast with GradScaler
• Track down a shape mismatch by printing shapes in the model forward or using einops/rearrange for explicit reshaping
• Find the source of NaN loss using anomaly detection, gradient checks, small lr, eps safeguards, and gradient clipping
• Fix device/dtype errors by moving model and inputs to the same device, using map_location when loading checkpoints, and casting tensors to the correct dtype
• Debug DataLoader crashes by lowering num_workers, using persistent_workers, and running with spawn start method in multiprocessing