News

vLLM adds opt-in FlyDSL MonoKernel decode for five models

vLLM has added opt-in FlyDSL MonoKernel decode paths for five models on selected AMD Instinct setups, with strict fallbacks and limits on its own performance and accuracy tests.

D
Nov 19, 2026 · 3 min read

vLLM has merged opt-in FlyDSL MonoKernel decode paths into its main branch for five models on selected AMD Instinct GPU configurations. The path targets low-concurrency generation latency. It is specific to certain models and hardware, remains off by default and has not been tied to a stable vLLM release containing all five implementations.

The merged support described by vLLM covers DeepSeek-V4.1-Flash, DeepSeek-V4-Pro, MiniMax-M3, Kimi-K3 and Muse Spark on MI355X. DeepSeek-V4.1-Flash also has a merged MI325X path. Operators activate an eligible setup with --monokernel; vLLM says an unsupported deployment is rejected at startup with the reasons listed instead of silently taking the specialized path.

Here, a MonoKernel combines several dependent stages within one persistent GPU-kernel boundary, allowing thread blocks to reuse resources, prefetch weights and hand off ready work without waiting at every whole-kernel boundary. That does not mean the entire model runs in one kernel. FlyDSL compiles an explicitly written schedule rather than automatically discovering or fusing the stages.

The integration is limited to eligible decoder-layer work. vLLM continues to handle scheduling, KV-cache management, sampling, speculative decoding and HIP-graph replay. Prefill, mixed batches and decode steps beyond a model’s supported width remain on the stock layers. In practice, the feature accelerates only configurations for which a specialized package has been implemented.

DeepSeek-V4.1-Flash has the broadest matrix. On MI355X, its merged paths support tensor parallelism, or TP, of 2 and 4, plus a TP8 setup positioned for concurrency 1. The MI355X step-width ceiling is 48 rows. The gfx942 port is benchmarked on MI325X at TP4 with a 42-row ceiling. Although the merged gfx942 port also names MI300X, its authors say copied weights use about 170 GiB per rank and leave little room on the 192 GiB accelerator; the project’s supported-configuration summary names MI325X rather than MI300X.

The other paths are narrower. DeepSeek-V4-Pro is specialized for TP8 across eight MI355X GPUs. MiniMax-M3-MXFP4 uses TP4 on MI355X and accelerates only the feed-forward half of layers 3 through 59 for one to 16 decode rows. Kimi-K3 targets TP8 on MI355X with DCP1 or DCP8 and an FP8 KV cache, with accepted concurrency and speculative-token combinations constrained to supported verify widths. Muse Spark Open MXFP4 uses eight MI355X GPUs with TP8, expert parallelism and a seven-token DSpark draft, covering decoder layers 1 through 60 for verify steps of one to 16 requests.

The performance results come from the implementation authors, not independent testing. Across six merged pull-request evaluations with same-node stock baselines, the project reports concurrency-1 improvements of 1.20x to 2.26x. DeepSeek-V4-Pro is outside that range because its comparison uses a historical InferenceX run on an older software stack; its pull request says the reported gains therefore do not isolate the kernels. AgentX results use one 3,600-second run per point.

The reported gains also shrink as concurrency rises. For MiniMax-M3 on random 1,000-token prompts and outputs, the authors report a time-per-output-token improvement of 16.6% at concurrency 1 and parity at concurrency 16. For DeepSeek-V4.1-Flash, the TP8 implementation is explicitly a concurrency-1 option: at concurrency 4, the authors report one TP8 server delivered 26% less throughput than two TP4 replicas.

The accuracy results have similarly narrow configuration boundaries. The authors characterize mono-on versus mono-off differences as within noise for specified DeepSeek-V4.1-Flash TP2 and MI325X tests and for specified MiniMax-M3, Kimi-K3 and Muse Spark setups. Those results do not establish equivalence for other GPU, TP or step-width combinations. DeepSeek-V4.1-Flash TP8 remains unresolved: its merged pull request reports GSM8K scores of 0.910 with mono decode and 0.923 without it across two runs, and says the gap was unexplained; the project post says accuracy for that setup was still being evaluated.

GLM-5.2 is not among the five merged paths. Its pull request was still a draft when the research was completed, while ports for GLM-5.x, Qwen3.8 and MiniMax-M3.1 were described as under review.

More news