Amazon makes SGLang-Neuron available for serving models on AWS Trainium

SGLang-Neuron is an out-of-tree plugin that connects SGLang to Trainium. After installation, setting SGLANG_PLATFORM=neuron selects the platform when the server starts, while SGLang core keeps handling the API server, request lifecycle, RadixAttention and prefix-cache bookkeeping. The plugin supplies the Trainium-specific models, attention, mixture of experts layers and graph runner.

Model execution uses standard PyTorch operations plus kernels written in the Neuron Kernel Interface (NKI), which torch.compile turns into executable programs. The plugin also supports PD disaggregation, in which a prefill server sends the KV cache to a decode server over NIXL on Elastic Fabric Adapter (EFA).

The walkthrough runs on a Trainium3 Amazon EC2 instance inside a Neuron container. It launches openai/gpt-oss-120b with tensor and expert parallel sizes of 8, then calls the standard chat completions endpoint. Setup steps and the list of supported models are in the SGLang-Neuron getting started guide.

Teams already running SGLang can try Trainium for serving or RL rollouts while keeping the same generation APIs in their code. Because the plugin is developed separately from SGLang core, Trainium support can move forward without waiting on core changes. Check the validated configurations and supported models before assuming a given model will run.