News

vLLM adds day-zero support for Inkling 2 Small

vLLM added serving support for Thinking Machines Lab’s Inkling 2 Small using custom decode and prefill kernels, DSpark speculative decoding and bounded LoRA support. It verified two GB200 configurations; the throughput and latency figures are internal, configuration-specific results.

D
Nov 19, 2026 · 3 min read

vLLM has added day-zero serving support for Thinking Machines Lab’s Inkling 2 Small, with model-specific kernels, speculative decoding and LoRA support for the newly downloadable open-weight model. That work gives operators a documented vLLM serving path for a 594-billion-parameter mixture-of-experts model whose attention, convolution and mixed-precision design required architecture-specific work.

vLLM’s documented compatibility evidence covers Nvidia GB200 systems with the bundled DSpark draft model and reasoning parser enabled. Its serving recipe lists one four-GPU tray using tensor parallelism 4 for contexts up to 262,144 tokens, and two trays using tensor parallelism 8 for the full 1,048,576-token context. The opened evidence does not establish compatibility outside those configurations.

At the research cutoff, the recipe required vLLM 0.35.0 or newer but described that version as not yet released. In the meantime, it directed operators to a nightly build or model-specific container.

Inkling 2 Small, according to its model card, is a 48-layer, decoder-only multimodal model that accepts text and images and produces text. It has 594 billion total parameters, activates 17 billion per token and routes each token through eight of 384 experts. The first 40 layers use NVFP4 experts, the final eight use MXFP8 experts, and the model supports contexts of up to 1 million tokens. Thinking Machines Lab released the weights under Apache 2.0 and made them available for download on November 12, the date vLLM published its integration post.

For decode, vLLM fused an all-reduce, short convolution, residual update, attention-residual mixing and RMS normalization into an allreduce_conv_mix kernel. The project says the kernel uses NVLink multicast and keeps post-collective work in registers. It also applied programmatic dependent launch to overlap parts of matrix multiplication, attention, routing and activation kernels.

The implementation also matches the checkpoint’s mixed-precision expert layout, folding activation quantization into routing and activation-producing kernels. vLLM says the expert matrix multiplications use TensorRT-LLM batched-GEMM kernels distributed through FlashInfer. For large prefills, it uses sequence parallelism so each GPU processes its token slice between a reduce-scatter and an all-gather instead of repeating that work on every rank.

Inkling 2 Small’s 6:1 grouped-query-attention ratio led vLLM to build a warp-specialized Gluon kernel for decode and speculative verification. According to the project, its existing FlashAttention-4 decode path did not pack this layout, while the new kernel packs six query heads into one tile and reads the key-value data once. The checkpoint includes a five-layer DSpark draft model with an RNN-style head that proposes seven tokens per step; vLLM’s documented command enables probabilistic drafting.

LoRA support has a checkpoint-specific limit. The recipe says adapters can target attention projections and routed experts, but expert adapters require BF16 expert weights. Operators using the released NVFP4 and MXFP8 experts are therefore limited to attention-only adapters.

The performance figures are vLLM’s own results, not independent benchmarks or general compatibility guarantees. In an 8,000-token prefill measurement, the project says sequence parallelism reduced the prefill step by 22% at tensor parallelism 4 and 26% at tensor parallelism 8, while freeing 1.3 to 1.9 GiB of memory per GPU for key-value cache. It also reports that its custom relative-attention kernel cut a batch-one decode step at a 100,000-token context from 15.1 milliseconds to 4.8 milliseconds.

For one user at tensor parallelism 4, vLLM reports about 630 output tokens per second with a speculative acceptance length of 3.5. In a two-node, tensor-parallelism-8 agentic workload with probabilistic DSpark drafting and prefix caching, the project reports about 1,070, 915, 700 and 450 output tokens per second per GPU at session contexts of 32K, 64K, 128K and 256K, respectively. Those figures were measured at concurrency levels where each user received 100 tokens per second, with roughly 3,000 new input tokens and 1,024 generated tokens per turn at temperature 1.0. The post does not provide a complete hardware and software reproducibility matrix for every microbenchmark.

vLLM says it checked its DSpark-enabled implementation on needle-in-a-haystack cases up to 1 million tokens, MMMU-Pro and Humanity’s Last Exam with the public Inkling Verify harness, and matched its own non-speculative runs. No independent reproduction was cited. The recipe also says the Inkling tool-call parser has unit tests, but no end-to-end tool-calling benchmark had been run.

More news