NVIDIA opens deterministic failure attribution and guarded recovery for training clusters
NVIDIA released Apache-licensed Attribution and Recovery components for distributed AI workloads. Attribution correlates scheduler state, logs and metrics into evidence-backed diagnoses; Recovery monitors Slurm jobs, records actions and can progress from observe-only recommendations to acknowledged automated recovery.
Why it matters: Long training runs need a traceable path from symptom to node isolation or restart. Separating diagnosis from the control loop makes autonomous remediation reviewable and lets teams begin in monitor mode.