AI Agent News/stories/aws-adds-managed-nvidia-dynamo-to-sagemaker-hyperpod-inference.md

Infrastructure

AWS adds managed NVIDIA Dynamo to SageMaker HyperPod Inference

Amazon SageMaker HyperPod Inference now supports NVIDIA Dynamo, an open source distributed inference framework with multi-node serving, disaggregated prefill and decode, and KV-aware routing. The add-on installs the Dynamo components and offers two deployment paths, bring-your-own DynamoGraphDeployment manifests or InferenceEndpointConfig with dynamoSpec.

Source: aws.amazon.com/blogs/machine-learning/managed-nvidia-dynamo-on-amazon-sagemaker-hyperpod-inference

AWS says large models such as GLM-5.3 can exceed the GPU memory of a single ml.p5.48xlarge instance, and long prompts can slow token generation when prefill and decode share GPUs. The HyperPod Inference add-on installs the Dynamo custom resource definitions along with the Dynamo operator, Grove for grouping multi-node pods, and the KAI Scheduler for gang scheduling. Dynamo runs with vLLM, SGLang, or TensorRT-LLM, with vLLM as the default, and AWS Deep Learning Containers supply the runtime.

Users can deploy either by writing a complete DynamoGraphDeployment manifest or by creating an InferenceEndpointConfig with a dynamoSpec section, which keeps the OpenAI-compatible interface and lets the operator handle endpoint setup, weights and autoscaling. HyperPod monitors node, GPU and EFA health and replaces failed nodes.

In AWS benchmarks on Qwen3-235B-A22B-FP8 using two p5en.48xlarge instances, disaggregated serving reached 207 output tokens per second per GPU against 101 for aggregated serving, at the targets the team set. For Qwen3.5-397B-A17B-FP8 split across two instances, a two-node aggregated setup kept 99.1 percent of single-node throughput, though the team noted this model fits on one instance and was used as a baseline.

Why it matters if you build agents

Teams serving models too large for one instance can use multi-node and disaggregated deployments without building the scheduling and health monitoring themselves.

Benchmark results come from AWS and NVIDIA on specific hardware and workloads, so developers should replay their own traffic with AIPerf before choosing a topology.

Multi-node deployments need EFA transport confirmed for KV-cache transfers between workers, so check networking before running benchmarks.

Get stories like this every weekday

One short email with the day's agent tooling news.

Subscribe