Speculative Decoding: Get 2-3x Faster Inference Without Better Hardware

Speculative decoding is one of the most elegant optimizations in modern LLM inference. By using a small "draft" model to speculatively generate multiple tokens at once, then verifying them with the large target model in parallel, you can achieve 2-3x speed improvements with zero quality loss. Here's how it works and how to enable it.

Hardware requirement: Speculative decoding needs enough VRAM to hold both the draft and target model simultaneously. For a 13B target model with a 1B draft model, you need roughly 9GB VRAM. See hardware requirements at LLMHardware.io's VRAM calculator.

How Speculative Decoding Works

The draft model (small, fast) generates a sequence of K candidate tokens. The target model (large, slow) then verifies all K tokens simultaneously — because verification is parallelizable while generation isn't. Accepted tokens are kept; the first rejected token causes the sequence to be truncated and re-generated. When the draft model is right (acceptance rate 70-80%+ for good draft/target pairs), you generate multiple tokens per target model pass instead of one.

Enabling in llama.cpp

llama.cpp supports speculative decoding via the llama-speculative binary. Command: ./llama-speculative -m target.gguf -md draft.gguf --draft 8 -n 256. The --draft 8 flag sets how many tokens the draft model generates ahead. Good draft models: use the base (non-instruct) version at 1/4 to 1/8 the size of your target model. Llama 3.2 1B works well as a draft for Llama 3.1 8B.

Self-Speculative Decoding

A newer variant called self-speculative decoding uses early exit layers from the same model as the draft. This eliminates the need for a separate draft model and works with any single model. Medusa heads add auxiliary prediction heads to the target model that speculate future tokens. Both approaches are available in more recent llama.cpp builds and offer 1.5-2x speedups without dual-model VRAM requirements.