What changed

AWS launched a HyperPod Global Inference Gateway that routes one OpenAI-compatible endpoint across clusters and Regions using capacity, queue, proximity and error signals. It also added managed Slurm extensions and EKS or Slurm observability for GPU, network, job and cluster diagnostics.

Why it matters

Routing, failover, software convergence and GPU fault visibility become managed control-plane functions instead of separate operator-built systems, changing both resilience and lock-in decisions for self-hosted inference.

What to do

Canary two Regions with synthetic traffic, fail one cluster, exhaust KV headroom and inject a GPU or EFA fault. Verify route choice, IAM and PrivateLink policy, extension convergence, alerts, recovery time and rollback.

Caveat

AWS and a named beta customer supply the evidence; independent latency, failover and diagnostic-quality results are absent. The gateway spans 13 Regions and extensions still incur charges for underlying observability resources.