speculative-decoding

Load a small draft model alongside your target model and pass it as assistant_model to the generate method in transformers 4.36+. The draft model proposes tokens that the target model verifies in parallel, giving 1.5-2× speedup with zero quality loss.

Medusa achieves 2-3.6× speedup by adding prediction heads to a single model but requires minimal training. Draft model speculative decoding needs no training and works if you already have a smaller version of your model, but delivers lower speedup.

No, speculative decoding is mathematically equivalent to standard target model decoding. Tokens are only accepted when they match the target model's probability distribution, so outputs remain identical in quality.

Select a draft model 5-10× smaller than the target model, such as Llama-2-7b as draft for Llama-2-70b. If your target model is too small for a useful draft, use Medusa or Lookahead Decoding instead.

Yes, vLLM supports speculative decoding through the speculative_model and num_speculative_tokens parameters when initializing the LLM class. This enables batched speculative generation for high-throughput serving.

Lookahead decoding achieves lower speedup (1.5-2.3×) than Medusa and has verification overhead that grows with window size. Larger models require smaller window and n-gram parameters because verification becomes more expensive.