# Amazon makes SGLang-Neuron available for serving models on AWS Trainium

published: 2026-11-12T22:05:38.284Z
topic: infrastructure
source: https://aws.amazon.com/blogs/machine-learning/deploy-frontier-ai-models-on-aws-trainium-with-sglang-neuron/
url: https://aiagentnews.io/stories/amazon-makes-sglang-neuron-available-for-serving-models-on-aws-trainium

Amazon has made SGLang-Neuron publicly available, letting developers serve models and generate reinforcement learning rollouts with SGLang on AWS Trainium. The plugin ships in an AWS Deep Learning Container released by RadixArk in collaboration with Annapurna Labs.

SGLang-Neuron is an out-of-tree plugin that connects SGLang to Trainium. After installation, setting SGLANG_PLATFORM=neuron selects the platform when the server starts, while SGLang core keeps handling the API server, request lifecycle, RadixAttention and prefix-cache bookkeeping. The plugin supplies the Trainium-specific models, attention, mixture of experts layers and graph runner.

Model execution uses standard PyTorch operations plus kernels written in the Neuron Kernel Interface (NKI), which torch.compile turns into executable programs. The plugin also supports PD disaggregation, in which a prefill server sends the KV cache to a decode server over NIXL on Elastic Fabric Adapter (EFA).

The walkthrough runs on a Trainium3 Amazon EC2 instance inside a Neuron container. It launches openai/gpt-oss-120b with tensor and expert parallel sizes of 8, then calls the standard chat completions endpoint. Setup steps and the list of supported models are in the SGLang-Neuron getting started guide.

## Why it matters if you build agents

Teams already running SGLang can try Trainium for serving or RL rollouts while keeping the same generation APIs in their code. Because the plugin is developed separately from SGLang core, Trainium support can move forward without waiting on core changes. Check the validated configurations and supported models before assuming a given model will run.