News

AWS adds EKS training checks and Slurm hung-job recovery to SageMaker HyperPod

AWS has added EKS-based training validation and Kubernetes diagnostics, plus opt-in Slurm hung-job detection that can report or cancel stalled work for requeueing.

D
Nov 18, 2026 · 4 min read

AWS has added three reliability capabilities to Amazon SageMaker HyperPod: training validation and Kubernetes resource monitoring for clusters orchestrated by Amazon EKS, and hung-job detection for clusters orchestrated by Slurm.

The new HyperPod controls cover different failure modes and take different actions. The EKS validation can mark instances as failed before workloads use them, while the EKS monitoring agent only reports cluster problems. On Slurm, the detector always emits an event and can, when explicitly configured, cancel stalled work so Slurm can requeue it.

EKS checks training and cluster components

MultiNodeTrainingValidation is a deep health check for supported EFA-enabled NVIDIA GPU instances in EKS-based HyperPod clusters. It runs a fixed Megatron-LM workload first on every instance and then across instances of the same type. An instance fails if loss becomes NaN or infinite, or if training does not finish within the time limit. AWS’s launch post says operators can add the check when instances are created or updated, or start it on demand for an instance group or selected instance IDs.

The check labels an instance as failed, but recovery is conditional: automatic node recovery must be enabled for HyperPod to reboot or replace it. A pass establishes only that the hardware trained AWS’s reference workload correctly; AWS says it does not guarantee that a customer’s own job will converge.

AWS said its test on two ml.p5.48xlarge instances took 1 hour and 28 minutes for the full deep-health-check sequence, with the two training-validation tests adding about eight minutes. AWS said instances could not accept workloads until the checks completed. The company also reported that the reference workload’s loss fell from 11.57 to 3.62 over 300 iterations, with no NaN or skipped iterations, on 16 H100 GPUs. Those figures are AWS test results, not independent benchmarks.

The separate HyperPod Kubernetes Resource Monitoring Agent runs continuously as one pod per EKS cluster and prefers CPU nodes. It watches node-condition and resource changes, failures in critical infrastructure pods and required DaemonSets, and failures in supported training jobs. For jobs that opt in, it can also read NCCL communicator state.

The agent reports findings as Kubernetes events and structured CloudWatch logs. It does not reboot nodes or restart jobs because the fault could be in the workload. Adding it to an existing cluster requires kubectl and Helm access and the HyperPod Helm chart. AWS warns that a chart upgrade also updates other chart-managed HyperPod components and recommends a server-side dry run to inspect the changes.

NCCL monitoring requires NCCL 2.24 or later and job-level opt-in. NVIDIA’s RAS documentation says the subsystem can identify outliers such as unresponsive processes and listens on localhost port 28028 by default. Rebinding that listener to the pod network lets the HyperPod agent reach it but creates a security exposure; AWS recommends a NetworkPolicy that limits TCP port 28028 to the monitoring agent while preserving traffic among the training pods.

Slurm detector can notify or cancel

The Slurm capability runs an agent on every node but is off by default. A job is monitored only when it sets HYPERPOD_HANG_RULES to a rules file that every compute node can read at the same path. AWS’s hung-job documentation says default detection watches job output and GPU activity; custom log-pattern rules can instead flag missing progress or matching errors.

Custom patterns replace default detection for that job. They can miss hangs the patterns do not describe, while short timeouts can misclassify normal pauses such as checkpoint saves. AWS recommends beginning with the default notify action while tuning rules.

Detection always writes an event to CloudWatch. With notify, the job keeps running. With an explicitly configured cancel action, the agent cancels the job and Slurm can return it to the queue. Requeueing remains conditional on the cluster’s MaxBatchRequeue limit; a job submitted with --no-requeue stays canceled.

In an AWS test on two ml.p5.48xlarge instances, the default detector flagged a synthetic hang 15 minutes after the job’s last output, and the requeued run started about 2.5 minutes later. AWS presented those measurements as test timings, not service-level guarantees.

AWS also modeled an eight-hour hang across 32 ml.p5.48xlarge instances as 256 wasted instance-hours, or 2,048 GPU-hours. Using an approximate on-demand rate of $66 per instance-hour in US East (N. Virginia), AWS calculated about $16,900 in cost without detection and about $530 with a 15-minute rule, for roughly $16,400 in avoided cost. The region-specific rate and scenario are AWS inputs to a hypothetical example, not independently verified savings.

The Slurm feature requires a HyperPod Slurm AMI released on or after November 10, 2026. Updating an older cluster replaces every instance’s root volume, so AWS says data stored there must be backed up first. The detector does not verify ownership of a rules file; AWS advises restricting write access to both the file and its parent directory because anyone who can alter or replace it can change detection behavior or trigger cancellation for every job that uses it.

More news