AWS bundles Ray LLM stack for multi-node inference on EKS

AWS introduced multi-node Ray LLM Deep Learning Containers for Amazon EKS, bundling the components needed to run one large language model across separate GPU nodes behind an OpenAI-compatible endpoint. The container stack targets models that do not fit on one node and deployments that split prompt processing from token generation.

The release expands AWS’s inference tooling alongside its SageMaker inference skill for coding agents, while targeting Kubernetes-based model serving rather than agent-guided managed deployment.

In AWS’s worked example, Qwen3.5-9B runs across two g6.8xlarge GPU nodes. One handles prefill, processing the prompt and building the model’s key-value cache. The other handles decode, reading and extending that cache as it generates tokens. Ray Serve LLM’s build_pd_openai_app helper takes separate configurations for the two phases and exposes them through one endpoint, without a custom request handler in the example.

The Ray LLM container stack uses Ray Serve LLM as the serving layer and vLLM as the inference engine. NIXL moves the key-value cache between the prefill and decode workers. AWS’s EFA stack provides libfabric and the AWS OFI NCCL plugin for inter-node communication.

In the EKS deployment, KubeRay creates a CPU-only Ray head and GPU worker pods, then Ray Serve places the prefill and decode deployments on those workers. The manifest requests one GPU and one EFA interface per worker, selects the efa provider, mounts shared memory and runs the workers privileged so NIXL can register GPU memory with the network interface. Both workers must stay in one Availability Zone because EFA traffic cannot cross Availability Zones or VPCs.

For both phases, AWS explicitly configures NIXL’s LIBFABRIC backend and warns that a fallback transport may run at TCP speed. The company recommends checking worker logs to confirm that libfabric was selected. Its container documentation lists Ray 2.58.0, vLLM 0.26.0, PyTorch 2.11.0, CUDA 13.0.2, Transformers 5.14.1 and Python 3.13 in the serving image.

AWS says it validates the bundled components together before release, applies security patches during the build and selects compatible CUDA, framework and serving-layer versions. The company did not publish the underlying compatibility test matrix or results in the opened materials.

AWS also says disaggregating prefill and decode can keep long-prompt work from delaying active token generation at scale. The launch post contains no independent latency, throughput or cost comparison with colocated inference. AWS separately reports that its cache settings cut Qwen3.5-9B weight-loading time on restart from about 145 seconds to about 2 seconds on g6.8xlarge. The materials do not disclose repetitions or variance, and the result has not been independently reproduced.