AI Agent News/stories/aws-adds-a-global-inference-gateway-for-routing-across-sagemaker-hyperpod.md

Infrastructure

AWS adds a Global Inference Gateway for routing across SageMaker HyperPod clusters

AWS has launched the Global Inference Gateway for Amazon SageMaker HyperPod, a single managed endpoint that routes requests across every cluster a team registers, in one Region or several. It selects the best cluster for each request and moves traffic off a failed cluster automatically, without any client changes.

Source: aws.amazon.com/blogs/machine-learning/introducing-amazon-sagemaker-hyperpod-global-inference-gateway-one-endpoint-across-regions-and-clusters

The gateway works one tier above the existing Inference Gateway, which already routes requests to the best pod within a single cluster. The new tier chooses a cluster and carries the request over the AWS backbone, with each call signed by SigV4 and authorized through IAM. Clients call one endpoint rather than tracking one per model or per cluster.

For each request, the gateway scores every healthy cluster on KV cache headroom, queue depth, Region proximity and error rate. Users can set routing weights for each signal, and the defaults give Region proximity the largest share at 0.40. Traffic reaches the clusters only through AWS PrivateLink, and the gateway does not log or store prompts or completions.

Setup has two steps: enabling globalRouting in each cluster's InferenceGatewayConfig, then calling CreateGlobalInferenceGateway with the clusters to include. Clusters can later be added or detached with AWS CLI commands, and the gateway is available in 13 AWS Regions.

Why it matters if you build agents

Teams that spread GPU capacity across clusters and Regions can drop the load balancers, DNS and cross-Region networking they would otherwise build to handle peaks and failover. Each published cluster runs a Network Load Balancer that is billed even when the cluster is not in any gateway. If JWT checks are used, enable them on every cluster that serves a model or on none, so a request does not succeed on one cluster and fail on another.

Get stories like this every weekday

One short email with the day's agent tooling news.

Subscribe