SageMaker Inference adds GPU partitioning with NVIDIA MIG

GPU partitioning uses NVIDIA Multi-Instance GPU (MIG) to divide a physical GPU into independent partitions, each with a fixed share of compute and dedicated memory. Each model can run in its own partition, and isolation is enforced in hardware. The feature works only on inference component endpoints, on the P4d, P5, G7 and G7e instance families, in seven AWS Regions.

Developers declare a layout with AcceleratorPartitionLayouts on a production variant in CreateEndpointConfig, then choose a partition type for each inference component with AcceleratorPartitionType. The layout applies to every accelerator in the variant, and no new API operations or IAM actions are added. AWS says carving accelerators added roughly 17 seconds to provisioning in its tests.

In AWS's comparison using Llama-3.1-8B-Instruct, one mig-3g.40gb partition of an H100 was measured against a whole A100-40GB. The partition delivered 53 to 89 percent more output tokens per second at the load where its cost benefit was largest, and it was 18 to 34 percent cheaper per million output tokens above a crossover concurrency, while the whole A100 was cheaper below it.

Teams can host several small models on one endpoint with hardware isolation, without running their own Kubernetes MIG stack. Partition types are fixed for the life of a component, and a request for a profile the variant did not declare is rejected rather than falling back to a whole GPU. Serverless, asynchronous and heterogeneous instance pool endpoints are not supported.