News

AWS turns on continuous SageMaker inference monitoring by default

SageMaker AI now continuously checks eligible generative AI inference endpoints for latency drift, idle GPUs and configuration gaps, with findings sent through AWS Health and production changes left to customers.

D
Nov 20, 2026 · 2 min read

AWS has made monitoring the default for eligible Amazon SageMaker AI generative AI inference endpoints, shifting optimization from a point-in-time check to an ongoing stream of operational findings. The new SageMaker capability watches for rising response times, underused GPUs and serving configurations that may be missing available improvements.

No setup is required for eligible endpoints, and AWS says the monitoring carries no additional charge. Findings can arrive while an endpoint remains in service. The system uses operational signals including latency, GPU utilization and serving configuration, but AWS says it does not collect the requests or responses handled by the endpoint.

SageMaker’s documentation identifies five types of findings: latency drift, underutilized GPUs, supported speculative decoding that is not enabled, a stale serving container and untuned GPU kernels. To detect latency drift, SageMaker compares the endpoint’s median time to first token or median end-to-end latency with the previous week and the endpoint’s own history. Customers do not have to set a threshold.

AWS Health delivers the findings in the console and by email. A single region-specific notification groups all findings for an endpoint and explains what SageMaker observed, the potential effect and a recommended action. The notification is also available as an AWS Health event in Amazon EventBridge, so customers can route it into an existing operations workflow.

The recommended action opens the SageMaker AI agentic experience in SageMaker Studio, with the affected endpoint and its recent traffic already filled in. Customers can run Generative AI Inference Recommendations on separate compute, comparing the current configuration with alternatives on response time, throughput and cost per million tokens. The production endpoint continues serving during the tests. Benchmark jobs incur standard SageMaker AI compute charges or use an applicable machine-learning reservation.

Nothing deploys automatically. Customers review the configuration differences and measured results, then decide whether to apply a benchmark result or keep the current configuration.

The launch notice lists availability in US East (N. Virginia and Ohio), US West (Oregon), Europe (Ireland and Frankfurt), and Asia Pacific (Singapore and Tokyo). Eligible targets are real-time endpoints and inference components on real-time endpoints that use GPU instances and the vLLM inference engine. CPU, Inferentia and Trainium endpoints are excluded. Customers can opt standard endpoints out with the OPT_OUT_TRACKING environment variable; opting out endpoints that host inference components currently requires contacting AWS Support.

More news