AWS launches GPU-aware routing for SageMaker HyperPod

AWS launched SageMaker HyperPod Inference Gateway, a Kubernetes-native managed add-on for Amazon EKS that routes model requests using live GPU and model-serving state.

AWS says the per-cluster gateway is available in regions that offer the HyperPod inference add-on. It sits in front of model-serving pods and is designed to use accelerator capacity more efficiently without requiring changes to model servers or client applications.

The routing path starts with a Body-Based Router, which reads the model field in an OpenAI-compatible request. If the request uses a low-rank adaptation, or LoRA, adapter, the router resolves it to the base model and uses an HTTPRoute to reach the corresponding InferencePool. An Endpoint Picker then ranks eligible serving pods with configurable weighted scorers based on Prometheus metrics from the model servers.

Four signals target different sources of inference delay. Queue-depth scoring steers work away from pods with larger backlogs. KV-cache scoring avoids pods whose key-value memory — the stored attention state used during generation — is close to full. LoRA-affinity scoring favors a pod that already holds the requested adapter in GPU memory, avoiding an adapter swap. If no pod has the adapter loaded, AWS says the gateway falls back to the pod with the most available capacity.

Prefix-affinity scoring favors a pod likely to retain a reusable prompt prefix, reducing repeated prefill computation for workloads such as multi-turn conversations and document question answering. AWS documents default weights of 2 for queue depth, 2 for KV-cache utilization and 3 for prefix affinity. Operators can also score running-request counts and predicted latency, or replace the default Endpoint Picker configuration. The underlying Kubernetes InferencePool pattern also supports endpoint selection using queue, KV-cache and active-LoRA state.

For comparison, AWS has separately documented prefix-aware routing for SageMaker real-time endpoints. That routing path is distinct from the HyperPod gateway announced here.

In its own benchmark, AWS reported up to an 82% reduction in time to first token, illustrated by a drop from 4.4 seconds to less than 800 milliseconds. These are AWS-run vendor results, not independently replicated measurements. The company said it tested four models ranging from 8 billion to 235 billion parameters on p5.48xlarge H100 and g5 A10G instances, comparing the gateway’s default routing with Kubernetes round-robin across the same replicas.

In the same issuer-run tests, AWS reported throughput gains of 8% for a mixed-GPU Llama 3.1 8B workload, 50% for mixed-GPU Qwen3 32B and 12% for bursty Llama 3.1 70B traffic. AWS described throughput as comparable for bursty Qwen3 235B, a shared-prefix Llama 3.1 8B workload and a uniform Qwen3 235B fleet. It did not provide raw data, reproducible scripts, sample counts or confidence intervals, and the results have not been independently verified.

AWS documentation requires vLLM 0.9.2 or later or SGLang 0.3.5.post1 or later. Earlier vLLM releases cause the gateway to ignore KV-cache utilization without reporting an error, AWS warns, while earlier SGLang releases lack a required metrics flag and fail to start. Gateway endpoints also have no request-level authentication or authorization by default; operators must configure JWT authentication or rely on VPC and network controls.

The launch covers the Tier 1 per-cluster router. AWS described a separate Global Inference Router for cross-cluster and cross-region failover, global rate limiting and cost-aware traffic shaping as coming soon, without giving an availability date.