What changed
Amazon SageMaker HyperPod added multi-node training validation for EKS, a Kubernetes resource monitoring agent, and Slurm hung-job detection that can emit CloudWatch events and cancel stalled jobs for requeue.
Why it matters
Large jobs can consume thousands of GPU-hours while appearing healthy, so training correctness, cluster dependencies and forward progress now have managed checks beyond node hardware status.
What to do
Enable the checks on a staging cluster, inject a NaN loss, crash-loop a device plugin and stop log or GPU progress, then verify labels, CloudWatch evidence, cancellation, requeue and checkpoint recovery.
Caveat
Features differ across EKS and Slurm, hung-job detection is opt-in per job, the Kubernetes agent reports rather than repairs, and AWS savings examples are scenario calculations rather than independent production results.