AWS makes SGLang-Neuron publicly available for Trainium

AWS made SGLang-Neuron publicly available through an AWS Deep Learning Container on November 12. The public-preview release gives developers a documented route to serving models and generating reinforcement-learning rollouts on Trainium through SGLang’s existing interface.

Developed by RadixArk in collaboration with Amazon’s Annapurna Labs, the release lets developers launch a model through SGLang’s server interface and use the same generation APIs. An out-of-tree plugin adds Trainium support without changes to SGLang core. The project’s repository describes the release as a public preview whose features, APIs and supported models may change.

That division keeps request handling in SGLang and accelerator-specific execution in the plugin. SGLang core manages the API server, request lifecycle, RadixAttention and prefix-cache bookkeeping. SGLang-Neuron implements the Trainium-specific models, attention, mixture-of-experts support and graph runner. Installing the plugin registers a Neuron platform through SGLang’s entry points; setting SGLANG_PLATFORM=neuron selects it when the server starts.

The plugin runs through TorchNeuron, AWS’s native PyTorch backend. Model execution uses standard PyTorch operations, supplemented where needed by Neuron Kernel Interface kernels. The graph runner uses torch.compile to turn those operations and kernels into executable programs. AWS documents TorchNeuron as a public-beta backend for Trainium2 and Trainium3 that uses PyTorch’s PrivateUse1 device mechanism and supports eager execution and torch.compile.

AWS’s container walkthrough uses a Trainium3 EC2 instance and the openai/gpt-oss-120b model. The public ECR image is tagged v0.1.0-neuron-py314-sdk-2.33.0-ubuntu26.04; the repository says it includes SGLang, SGLang-Neuron, the Neuron SDK and a patched version of NIXL. Separate Trainium2 model compatibility records provide context on another effort to document model support for AWS accelerators.

SGLang-Neuron also supports disaggregated serving. One server prepares a prompt’s key-value cache, then transfers it to a separate decode server through NIXL over Elastic Fabric Adapter. Each server retains SGLang’s scheduler and request lifecycle, while the plugin manages Trainium execution and the cache-transfer path.

The project currently lists large-language-model support on Trainium3 instances and diffusion-model support on trn2.48xlarge. The opened materials provide no independent benchmark results, general-availability date, separate plugin pricing or region-by-region availability list.