If you’ve been running models locally, you already know the drill. Every token an LLM produces requires a full forward pass through every layer of the network. The model doesn’t get to guess ahead – it computes one token, appends it to the sequence, and starts the whole process over again. That’s the autoregressive pattern, and it’s the reason a 70B model can feel sluggish even on a card with plenty of headroom, as the BentoML engineering team explains.
We’ve spent a lot of time on this site talking about the VRAM side of that equation – choosing a quantization format, figuring out what the 32GB threshold unlocks, and how far you can push a 70B model on a single RTX 5090. Those posts are about fitting bigger, smarter models into limited memory. Speculative decoding attacks a different bottleneck: even once the model fits, generation is still gated by sequential compute, one token at a time.
The Trick: Let a Small Model Do the Guessing
Speculative decoding pairs two models instead of one. A small, fast “draft” model proposes several tokens ahead – three, four, sometimes more in a single burst. The large “target” model then checks all of those proposed tokens in one parallel forward pass, accepting the ones that match what it would have generated on its own and throwing out the rest, as described in BentoML’s LLM Inference Handbook.
This is the part that matters most for anyone worried about quality: the target model is still the one making every real decision. The draft model doesn’t get a vote – it gets a guess. If the guess is wrong, the target model rejects it and falls back to generating that token itself. NVIDIA’s technical blog on the topic frames this as a rejection-sampling scheme that keeps latency down without touching output quality – the verification step is what makes the whole thing safe to trust. NVIDIA’s own writeup notes acceptance rates in the 60-80% range for well-matched draft/target pairs, which is the difference between a speedup that’s real and one that’s marginal.
That’s a different value proposition than most of the optimization techniques we’ve covered before. Quantization trades a small amount of numerical precision for VRAM savings. KV cache quantization does something similar for context length. Both are genuine tradeoffs – you’re giving something up to get something back. Speculative decoding doesn’t ask you to trade away output quality. According to Redis’s writeup on the technique, speculative decoding makes responses faster without changing what the model outputs at all. While it still requires additional resources and setup, it avoids the precision-for-performance tradeoff common in this field.
Why the Output Stays Identical
It’s worth sitting with why this works, because it sounds too good the first time you hear it.
The draft model isn’t replacing the target model’s judgment – it’s front-loading the arithmetic. Normally, generating five tokens means five sequential forward passes through the big model, each one waiting on the last. With speculative decoding, the small model proposes five tokens cheaply, and the big model verifies all five in a single parallel pass, since a transformer can score a batch of candidate tokens about as cheaply as it can score one. The LLM Inference Handbook describes this draft-then-verify pattern as a guarantee: the final output matches exactly what the target model would have produced generating token by token on its own.
If the draft model nails three tokens in a row, you just did the compute-equivalent of one forward pass instead of three. If the draft model whiffs on the first guess, your output quality is no worse than standard decoding – the target model generates that token itself, same as always, though you’ve paid the overhead of the failed guess. The downside case is bounded; the upside case is a genuine multiplier.
What the Speedup Actually Looks Like
The gains reported across the industry aren’t uniform, because they depend heavily on how good the draft model’s guesses are. A Medium writeup from AI Science points to roughly a 2x speed increase in their benchmarks, with acceptance rates around 50-60% using an off-the-shelf small model as the draft. BentoML’s own engineering post reports up to 3x gains when the draft model is well-matched to the target and trained specifically for the pairing, pushing acceptance rates closer to 80-90%.
The variance between those numbers is the whole story. Speculative decoding isn’t a fixed multiplier you get for free just by turning it on – it’s a function of how often the small model’s guesses survive verification. A draft model that’s badly mismatched to the target – different training data, different tokenizer behavior, wildly different capability – will get rejected constantly, and you’ll pay the overhead of running two models for barely any benefit. A draft model that’s trained to shadow the target closely will get long streaks of accepted tokens, and that’s where the 3x numbers come from.
This is the same lesson we keep running into with local inference generally: the 7B and 8B parameter range has become the practical sweet spot for standalone models because of the balance between capability and hardware cost. In a speculative decoding setup, that same class of small model often becomes the draft – not because it’s meant to stand alone, but because it’s fast enough to propose tokens for a 70B target without adding meaningful latency of its own.
Where This Fits Your Local Stack
If you’ve read our piece on the 32GB threshold, you know that once you have enough VRAM to load a 70B model at a reasonable quantization, the next constraint isn’t fitting the model – it’s how fast it generates. That’s exactly the gap speculative decoding is built to close.
Running two models at once does cost VRAM, though – you need room for the draft model’s weights and its own KV cache alongside the target model’s. This is where the quantization decisions from our Q4_K_M vs Q5_K_M vs Q8_0 comparison start to interact with speculative decoding directly. A heavily quantized draft model frees up headroom for a less-compressed target model, or vice versa. There’s no universally right split – it depends on what you’re optimizing for, same as the quantization tradeoffs we’ve written about before.
Picking a draft model is its own small research project. You want something in the same “family” as your target – same tokenizer, similar training distribution – because acceptance rates drop fast when the two models disagree about basic things like sentence structure or common phrasing. The Hugging Face Model Hub is the natural place to look for compatible pairs, since many model families ship both a full-size and a distilled or smaller variant trained on similar data. The Open LLM Leaderboard is useful context here too – not because leaderboard rank tells you acceptance rate, but because it gives you a sense of which small models are capable enough to make good guesses in the first place. A draft model that’s too weak will get rejected so often it stops paying for itself, and you’ll want to benchmark acceptance rate directly on your own prompts rather than trust a single published number.
The Honest Limits
Speculative decoding isn’t a universal accelerator. It shines on tasks where the target model’s output is fairly predictable token-to-token – structured code, common phrasing, repetitive formats. It helps less on tasks where the model is genuinely uncertain at every step, because uncertain generation means low acceptance rates, and low acceptance rates mean you’re paying the draft model’s overhead without collecting the reward.
It also adds a layer of operational complexity. You’re now managing two models, two sets of weights in memory, and a verification loop instead of a single decode loop. That’s a real cost in engineering time, even if it’s not a cost in VRAM or output quality. If you’re already juggling KV cache quantization settings and GGUF format choices to squeeze a 70B model onto a single card, adding a draft model is one more variable in a system that already has several moving parts.
That said, the fundamental trade here is unusually clean. Most of the optimization work we cover – undervolting, quantization, KV cache tricks – involves giving up something measurable to get something back. Speculative decoding is closer to a pure engineering win: extra complexity in exchange for real speed, with the output guaranteed to match what you’d have gotten anyway.
Where This Leaves Local Inference
The throughline across everything we’ve written about local LLMs – from the offline revolution piece to the VRAM math on the RTX 5090 – is that the constraints keep moving, not disappearing. First it was fitting a good model into consumer VRAM at all. Then it was fitting a large model without destroying it with aggressive quantization. Now that 70B-class models genuinely run on a single high-end card, the next constraint is raw generation speed, and speculative decoding is one of the more elegant answers available.
It won’t replace the work of picking the right quantization format or managing your KV cache – those are still the foundation. But once that foundation is solid, pairing a small, fast draft model with your large target model is a way to get more speed out of hardware you already own, without touching the thing that actually matters: what the model says.
For developers, the practical takeaway is simple: if you’ve already solved the VRAM problem and your 70B model fits comfortably, speculative decoding is very likely worth the setup cost, especially for code generation, structured output, or any workload where the target model’s next token is often predictable. Start by testing an existing small model from the same family as your target on the Hugging Face Model Hub before investing in a purpose-trained draft model – the acceptance rate you measure on your own prompts will tell you within an hour whether the 2x or 3x numbers are realistic for your use case.
For founders evaluating local inference as an alternative to API costs, the calculus is similarly direct: speculative decoding is one of the few optimizations in this space that improves latency without a quality tradeoff attached, which makes it low-risk to adopt once the underlying model choice and quantization format are already settled. The complexity cost is real, but it’s an engineering cost, not a product-quality cost – and that distinction is usually what determines whether an optimization is worth shipping.
Sources
- https://www.bentoml.com/blog/speculative-decoding-in-llm-inference
- https://bentoml.com/llm/inference-optimization/speculative-decoding
- https://developer.nvidia.com/blog/an-introduction-to-speculative-decoding-for-reducing-latency-in-ai-inference/
- https://redis.io/blog/speculative-decoding-for-faster-llm-inference/
- https://medium.com/ai-science-hub/speculative-decoding-explained-a-2x-speedup-for-llm-inference
- https://huggingface.co/models
- https://huggingface.co/spaces/open-llm-leaderboard/open_llm_leaderboard
