speculative-decoding_skill

Overview

This skill accelerates LLM inference using speculative decoding, Medusa multiple heads, and lookahead (Jacobi) decoding to reduce latency and increase throughput. It targets real-time and resource-constrained deployments, delivering typical speedups of 1.5–3.6× while preserving model quality. The content covers draft-model workflows, Medusa head training, lookahead parameters, and production deployment patterns.

How this skill works

Speculative decoding uses a small draft model to propose token candidates and a large target model to verify them in parallel, accepting tokens when probabilities align. Medusa augments a frozen base LLM with multiple prediction heads that predict future tokens in one forward pass, enabling tree-based verification without a separate draft model. Lookahead decoding reformulates autoregressive decoding as a parallel Jacobi iteration that generates and verifies n-gram candidates across a sliding window.

When to use it

• When you need 1.5–3.6× inference speedup without quality loss
• To cut latency for chatbots, code generation, or interactive agents
• When serving high-volume traffic on constrained hardware
• If you want to deploy faster generation without changing core model architecture
• When you prefer minimal training (Medusa heads) or plug-and-play methods (Lookahead)

Best practices

• Select a draft model 5–10× smaller than the target for best speculative gains
• Tune K (speculative tokens), posterior_threshold, and window/W/N parameters per model size
• Use Medusa for new deployments when you can add heads; freeze base LLM initially (Medusa-1)
• Apply Lookahead for zero-training, out-of-the-box speedups and simple integration
• Benchmark end-to-end quality and latency with realistic prompts before production rollout

Example use cases

• Real-time customer support chatbot requiring sub-second responses
• Code generation service where throughput and latency both matter
• Deploying large LLMs on limited GPU fleets to increase serving capacity
• Hybrid setups: Medusa as draft for speculative decoding to combine speed and final-model quality
• vLLM production pipelines using speculative_model + num_speculative_tokens for scale