AWS brings local model caches to SageMaker HyperPod inference

AWS has launched model caching for Amazon SageMaker Inference on HyperPod, allowing operators to preload model weights and inference container images onto cluster nodes before deployments need them. The feature is generally available in every AWS Region where SageMaker HyperPod is available, according to the company.

The change cuts repeated network downloads when an inference deployment scales out. Rather than fetch both assets again, new pods can read weights from local NVMe storage and use a container image already pulled onto the node. The weights and image caches are separate, disabled by default and independently configurable through modelCacheConfig on an InferenceEndpointConfig or JumpStart deployment.

For weights caching, the HyperPod Inference Operator downloads model files to eligible nodes, labels a node once its cache is warm and mounts the cached files read-only inside inference pods. AWS documentation says each deployment receives an isolated cache directory. Scheduling favors nodes with a ready cache, but a pod placed on an uncached node can still load weights from the remote source and pull its container image normally.

Image caching takes a separate route to the same goal. The operator pre-pulls the inference-server image onto target nodes, so a later pod start can skip a registry transfer when the image is already present. AWS says the caches support JumpStart deployments and custom models stored in Amazon S3 or Amazon FSx. Its launch post also lists models from Hugging Face Hub.

AWS reports that cached weights can be read from local NVMe at about 7 GB/s, allowing pods to begin serving in seconds rather than tens of minutes. In company benchmarks involving models from 57 GB to 145 GB, AWS says weights caching made scale-out about 60% faster. It also reports that image caching removed more than two minutes of cold image-pull time and cut that time by up to 97% compared with pulling a fresh image from Amazon ECR on each pod start. AWS did not publish a reproducible benchmark methodology in the announcement, and the results have not been independently validated.

The initial transfer still has to happen. A node must first download the model data, and a replacement node must warm its cache before receiving warm-node scheduling preference. Weights caching also requires an instance with enough local NVMe capacity at the configured host path; AWS says EBS-only instances are not supported. If files change at the same S3 path, the cache does not detect the update automatically. Operators must change the deployment specification, such as the model path or a version suffix, to load the new weights.

The HyperPod Inference release notes date operator version 3.3, which introduced host-local model caching, to August 4, 2026, more than a month before AWS’s September 10 announcement. For context on the service’s broader operational tooling, DataPhoenix has also covered the HyperPod InstantStart control plane.