What changed

Amazon SageMaker HyperPod added multi-node training validation for EKS, a Kubernetes resource monitoring agent, and Slurm hung-job detection that can emit CloudWatch events and cancel stalled jobs for requeue.

Why it matters

Large jobs can consume thousands of GPU-hours while appearing healthy, so training correctness, cluster dependencies and forward progress now have managed checks beyond node hardware status.

What to do

Enable the checks on a staging cluster, inject a NaN loss, crash-loop a device plugin and stop log or GPU progress, then verify labels, CloudWatch evidence, cancellation, requeue and checkpoint recovery.

Caveat

Features differ across EKS and Slurm, hung-job detection is opt-in per job, the Kubernetes agent reports rather than repairs, and AWS savings examples are scenario calculations rather than independent production results.