nanogpt_skill

Overview

This skill packages a compact, educational GPT implementation in roughly 300 lines. It reproduces GPT-2 (124M) training behavior on OpenWebText while remaining easy to read, modify, and run on modest hardware. The goal is hands-on learning: inspect the model, tweak configs, and run end-to-end training and sampling quickly.

How this skill works

The code implements a minimal Transformer decoder (multi-head attention, MLP, positional embeddings) and a straightforward training loop in pure PyTorch. It supports character-level and BPE-style workflows, data preparation, optional compilation for PyTorch 2.0, mixed-precision dtypes, and multi-GPU training via torchrun/DDP. Utilities handle dataset binarization, checkpoint loading (including GPT-2 checkpoints), and sampling with temperature/top-k.

When to use it

• Teaching or learning how GPT architectures and training loops work
• Prototyping transformer changes or research ideas with minimal overhead
• Running quick experiments on small datasets (Shakespeare) on CPU or modest GPUs
• Reproducing GPT-2 (124M) if you have multi-GPU resources
• Fine-tuning a pretrained GPT-2 checkpoint for domain-specific text

Best practices

• Start with the Shakespeare character workflow to verify environment and pipeline before larger runs
• Keep configs in the provided files and change only a few hyperparameters at a time
• Use gradient accumulation to simulate larger batch sizes when VRAM is limited
• Enable compile=True on PyTorch 2.0 for faster iteration and use bfloat16/float16 to reduce memory
• Monitor OOM by lowering batch_size or block_size and increase gradient_accumulation_steps to keep effective batch

Example use cases

• Train a toy Shakespeare model end-to-end on CPU to learn tokenization and sampling
• Reproduce GPT-2 (124M) on OpenWebText with 8× GPUs to study scaling behavior
• Fine-tune a GPT-2 checkpoint on a custom dataset for creative writing or domain text generation
• Experiment with alternative attention or MLP blocks by editing the ~300-line model file
• Build a teaching lab: students modify model hyperparameters and observe effects on training and samples