AWS adds GPU partitioning to SageMaker Inference

Amazon Web Services has added NVIDIA Multi-Instance GPU partitioning to Amazon SageMaker AI real-time inference components, letting multiple models run in hardware-isolated slices of a supported accelerator on a managed endpoint. Each slice receives a fixed share of GPU compute and a dedicated block of GPU memory, allowing teams to deploy models that do not need an entire accelerator.

The feature is available on P4d instances with NVIDIA A100 40 GB GPUs, P5 with H100 80 GB GPUs, G7 with RTX PRO 4500 32 GB GPUs, and G7e with RTX PRO 6000 96 GB GPUs. AWS lists seven supported regions: US East (N. Virginia and Ohio), US West (Oregon), Asia Pacific (Singapore and Tokyo), and Europe (Frankfurt and Ireland). Individual instance availability and endpoint quotas still vary by region.

NVIDIA’s Multi-Instance GPU technology, or MIG, divides a physical GPU into independent instances with isolated compute and memory paths. SageMaker exposes it through two optional fields: AcceleratorPartitionLayouts sets the layout for every accelerator in an endpoint production variant, while AcceleratorPartitionType assigns an inference component to one of the declared partition types. Invocation does not change; applications continue to address the inference component rather than the partition. AWS added no new API operations, resource types or IAM actions.

This approach differs from SageMaker multi-model endpoints. Multi-model endpoints use a shared serving container, dynamically downloading, loading, caching and unloading target models as requests arrive and memory fills. Infrequently used models can face cold-start latency. By contrast, MIG-backed inference components keep separately deployed models in fixed GPU partitions, avoiding model swapping while providing hardware isolation between GPU slices.

That isolation does not extend to the entire host. Models on the same instance still share CPU, host memory, storage, networking and endpoint routing, so GPU partitioning is not a complete tenancy boundary. The maximum number of partitions also depends on the accelerator: up to seven per A100 or H100, two per RTX PRO 4500, and four per RTX PRO 6000. A partition cannot span accelerators, leaving models that require a full GPU or multiple GPUs on whole devices.

AWS documents several operational constraints. Partitioning applies only to real-time endpoints that use inference components; single-model endpoints, serverless endpoints, heterogeneous instance pools and asynchronous invocation are unsupported. An endpoint’s partition layout cannot be changed while it is running, and an inference component’s partition type is fixed for that component’s lifetime. Changing either requires creating a new endpoint or component. A requested profile must already exist in the endpoint layout, with no fallback to a whole GPU.

CloudWatch supplies invocation and latency metrics for partitioned endpoints and their inference components, but AWS says they do not publish GPU, CPU or memory utilization metrics. Customers are billed for the full GPU instance rather than by partition, so partitioning can increase the share of an instance in use but does not reduce its hourly price.

AWS also published its own comparison of one H100 mig-3g.40gb partition with one whole A100 40 GB GPU. In tests using Llama-3.1-8B-Instruct, five traffic shapes and concurrency from 2 to 512, AWS said the H100 partition delivered 53% to 89% more output tokens per second at each scenario’s peak cost-benefit load. It also reported 18% to 34% lower cost per million output tokens above workload-dependent concurrency thresholds.

Those figures are vendor benchmarks, not independently reproduced general results. AWS calculated unit cost from full-instance on-demand prices in US West (Oregon), and said the whole A100 remained cheaper below the tested crossover points. The company advised customers to compare options on their own workloads. On P4d, AWS also warns that the mig-1g.10gb profile needs a newer driver than the default image supplies and requires the documented SageMaker inference AMI setting.