AWS details how HyperPod data capture feeds model workflows

AWS has detailed how SageMaker HyperPod customers can turn captured inference traffic into data for monitoring, debugging and model improvement. Its technical walkthrough connects live inputs and outputs stored in Amazon S3 with Athena analysis, labeling, fine-tuning and redeployment on a HyperPod cluster.

This is not a new launch. The post expands on a capability AWS introduced earlier in 2026. The company’s HyperPod Inference release notes date its introduction to May 6, while AWS announced broader availability on May 20.

Teams can enable HyperPod Data Capture independently at three points in the request path. Tier 1 records input and output payloads at the SageMaker endpoint and works with SageMaker Model Monitor. Tier 2 saves Application Load Balancer access logs, including request paths, client IP addresses and latency, but not request or response bodies. Tier 3 records payloads at the model pod, with controls for sampling, buffering and payload-size limits. All three tiers write to S3. A separate AWS infrastructure update added GPU partitioning for SageMaker real-time inference.

Configuration requires at least one tier in the dataCapture section of an InferenceEndpointConfig or JumpStartModel custom resource. Tier 1 and Tier 3 can use customer-managed AWS KMS keys. According to the documentation, Tier 2 relies on S3’s default encryption because Application Load Balancer access logs do not support KMS encryption.

AWS says capture delivery to S3 is asynchronous and does not block inference. In a synthetic test of 400 routing requests at concurrency eight, the company measured p99 latency of 249.4 milliseconds without capture and 249.8 milliseconds through the Tier 3 proxy. Both paths handled 39.1 requests per second. These are AWS’s measurements and were not independently reproduced.

For the walkthrough, AWS used synthetic support-ticket traffic to demonstrate a model-improvement loop. It queried captured records with Athena, applied a SQL labeling policy, fine-tuned the model and deployed a second endpoint. AWS reported that accuracy on a synthetic 280-item held-out set rose from 83.9% to 100%. On a separate 24-item hand-written probe, correct responses increased from 16 to 20. The exercise does not establish results for an external production deployment.

Capturing full payloads also creates data-governance obligations. AWS warns that records may contain personal information and recommends sampling, lifecycle policies, KMS encryption for Tiers 1 and 3, and redaction before the data is reused for training. Enabling both payload tiers stores the same traffic twice. AWS estimates that Tier 3’s three sidecars request a combined 250 millicpu and 640 MiB per pod, in addition to S3 storage and request charges.

AWS says the feature is available for EKS-orchestrated HyperPod clusters in every AWS Region where HyperPod is supported. AWS identifies S3 storage and requests, Athena scans and Tier 3 sidecar resources as potential cost categories.