AWS maps two vLLM KV-transfer paths to separate latency bottlenecks

AWS has documented two vLLM KV-cache transfer methods aimed at separate sources of latency in disaggregated inference. A push-based NIXL connector changes how cache state moves from prefill to decode, while bi-directional transfer can reuse decode-side state on later conversation turns. One moves transfer setup off the first-token path after prompt processing; the other avoids recomputing conversation history on subsequent turns.

Disaggregated inference splits prompt processing, or prefill, from token generation, or decode. Prefill builds the key-value cache that holds attention state, and decode uses that state to generate output. The split lets operators scale the two phases independently, but cache transfer becomes part of the wait for the first generated token.

The standard NIXL connector uses a pull flow: after prefill finishes, decode receives source metadata, allocates destination blocks, prepares the transfer and issues a NIXL read. In the push-mode design documented by vLLM, decode registers destination memory earlier. Prefill can then issue a NIXL write as soon as both the cache blocks and registration are ready. Each worker rank has a dedicated background writer thread for push-specific notifications and transfers, keeping descriptor preparation and transfer submission out of the engine’s latency-sensitive main loop.

AWS said push mode delivered up to a threefold improvement in time to first token and up to a 30% improvement in time per output token over pull mode. Its test used Qwen3-32B on a one-prefill, one-decode EC2 P5en deployment with Elastic Fabric Adapter, with prefix caching disabled to force a full cache transfer for every request. These are AWS benchmark claims and have not been independently verified. Push mode also requires an early decode selection so memory can be registered before prefill completes. AWS said pull mode preserves more flexibility to choose a decode worker afterward for load-aware routing.

Bi-directional transfer tackles repeated computation between turns. A stateful, cache-aware proxy uses a client-provided conversation_id to retain decode block-location metadata from one response and attach it to the next request. Prefill can then read reusable blocks from decode over RDMA and compute only the new tokens. The proxy keeps location metadata, not the KV tensors.

The vLLM NIXL connector guide sets a default 64-token recompute threshold. Below that threshold, prefill recomputes locally instead of transferring blocks. The guide also specifies a default, non-renewing 480-second retention period for KV blocks on decode. Both settings are configurable.

In separate tests, AWS said bi-directional transfer reduced time to first token by factors of up to 1.6 for Qwen3-32B and up to 3.3 for GLM-5-FP8 at the longest prompt lengths tested. The one-prefill, one-decode P5en setup used Elastic Fabric Adapter. AWS measured about 30 to 130 milliseconds of KV-transfer time for Qwen3-32B and 29 to 110 milliseconds for GLM-5-FP8, and said the avoided prefill computation outweighed that cost. The figures have not been independently verified. AWS also disabled prefill prefix caching to emulate eviction or routing to another prefill node.

The methods are distinct from AWS’s earlier Ray LLM container and Amazon EKS deployment stack. That work covered packaging and orchestration: Ray Serve behind an OpenAI-compatible endpoint, with separate prefill and decode GPU workers using NIXL over Elastic Fabric Adapter. The new posts instead focus on connector-level scheduling of KV movement and cross-turn cache reuse within that kind of disaggregated serving setup.