saelens_skill

Overview

This skill provides practical guidance for training, loading, and analyzing Sparse Autoencoders (SAEs) using SAELens to extract sparse, interpretable features from model activations. It explains workflows for loading pre-trained SAEs, training custom SAEs, and using features for analysis and steering. The focus is on reducing polysemanticity and revealing monosemantic representations in language models.

How this skill works

SAEs learn a sparse intermediate representation by encoding model activations into a higher-dimensional bottleneck with an L1 sparsity penalty, then decoding back to reconstruct the original activations. Training minimizes reconstruction error plus an L1 term so only a small number of SAE features activate per input. Once trained or loaded, SAE.encode yields sparse feature activations and SAE.decode reconstructs activations; feature directions can be attributed to logits or added as steering interventions.

When to use it

• Discover interpretable features in dense residual activations
• Analyze superposition and the geometry of feature representations
• Study monosemanticity or identify safety-relevant features (bias, deception)
• Perform feature-based steering or ablation experiments
• Validate model behavior with feature-level attribution

Best practices

• Match the SAE hook point to the model layer and residual stream you analyze
• Set d_sae to 4–16× d_model and tune L1 coefficient for desired sparsity
• Use an L1 warm-up (500–2000 steps) and ghost grads to avoid dead features
• Monitor L0 (active features), CE-recovery, dead-feature ratio, and explained variance during training
• Start with pre-trained SAEs for exploration before training large custom SAEs to save compute

Example use cases

• Load a pre-trained SAE to list top-activating features per token and inspect human-readable triggers
• Train a custom SAE on a target layer to discover domain-specific concepts (code syntax, legal language)
• Compute per-feature logit contributions to explain token predictions like named entities or dates
• Steer generation by adding decoded feature directions to the residual stream at generation time
• Diagnose training issues: reduce batch size for memory errors, lower L1 for better reconstruction, or switch to TopK to force exact sparsity