speculative-decoding

Speculative decoding uses a small draft model to generate candidate tokens that the large target model verifies in one parallel forward pass. In transformers 4.36+, pass the draft model as assistant_model to the target model's generate method for 1.5-2× speedup with no quality loss.

Medusa adds trained prediction heads directly to the base model, eliminating the need for a separate draft model and achieving 2-3.6× speedup. Draft model speculative decoding requires an external small model but needs no training, typically yielding 1.5-2× speedup.

Select a draft model 5-10× smaller than the target model from the same family for compatible tokenization. For example, use Llama-2-7b as the draft for Llama-2-70b; smaller targets under 13B benefit more from Medusa or lookahead decoding instead.

Lookahead decoding requires neither training nor a draft model. It reformulates autoregressive decoding as Jacobi iteration, generating and verifying n-grams in parallel to achieve 1.5-2.3× speedup out-of-the-box with any existing model.

Speculative decoding is mathematically equivalent to standard target-model sampling, so outputs have zero quality loss. Medusa-1 with a frozen backbone is also lossless, while aggressive Medusa-2 configurations may trade minor quality for higher speedup.

Yes, vLLM supports speculative decoding by passing speculative_model and num_speculative_tokens parameters when initializing the LLM class. This enables production serving of large models like Llama-2-70b with a smaller draft model for reduced latency.