Cohere details dynamic speculative decoding and contributes the method to vLLM

Cohere said its hardware-aware dynamic speculative decoding technique runs about 23% faster than conventional fixed-draft speculative decoding on its Command A model at large batch sizes. The company has contributed the method to the open-source vLLM inference framework.

Speculative decoding speeds up large-language-model inference by having a small “draft” model propose several tokens that the main model then verifies in a single pass. Its weakness is that a fixed number of draft tokens, K, hurts throughput once many requests run at once. Cohere’s dynamic speculative decoding, or DSD, adjusts K to the current batch size and hardware load rather than pinning it, which the company says avoids that high-concurrency regression.

In its technical write-up, Cohere reported the roughly 23% gain over fixed-K decoding at batch sizes of 128 and 256 on the dense Command A model, and a 7.5% gain over standard non-speculative inference at batch size 128. On the mixture-of-experts Command A+ model, DSD and ordinary speculative decoding delivered similar speedups, with DSD selecting a higher draft count — five draft tokens versus three — at mid-range batch sizes.

The post, written by Cohere technical-staff member Ekagra Ranjan, said the technique was submitted as a pull request to vLLM, with implementation notes for compatibility with asynchronous scheduling and CUDA Graphs.

The numbers are Cohere’s own and have not been independently verified; speculative-decoding gains vary widely with model, hardware and traffic pattern, so results on other setups may differ. Contributing the code to vLLM, which many teams use to serve models, gives outside engineers a way to test the claims directly.

Whether the vLLM maintainers merge the pull request, and whether independent benchmarks reproduce the 23% figure, will show how broadly the technique holds up.