SGLang 0.5.22 adds new model and accelerator paths, a 10x-faster InstantTensor loader claim, Apple Silicon support through PyTorch and multiple speculative-decoding optimizations. It also fixes a correctness issue where hybrid linear models could resume from the wrong recurrent state when structured output and speculative decoding were combined.
Why it matters: The release mixes throughput claims with a concrete correctness repair on a high-risk serving combination. Operators gain wider deployment options, but Python and runtime changes make a controlled upgrade more important than the headline performance numbers.
vLLM 0.32.0 adds a POST /v1/systemone endpoint for structured decisions, a generic Hugging Face tool-and-reasoning parser, lower-precision KV-cache paths and Model Runner V2 improvements. The release also restricts CORS by default and includes fixes for three published security advisories.
Why it matters: The release expands vLLM from text and tool serving into typed decision workloads while changing security and compatibility defaults. Teams can consolidate more inference behind one runtime, but upgrades can alter browser access, Python support, stream…
SGLang released signed version 0.5.21 with a low-latency /v1/decisions classifier and scorer API, /v1/score, on-the-fly switching between prefill and decode modes, and a Rust prefix cache enabled by default. The release also expands model, diffusion and AMD MI355X support and publishes NVIDIA, AMD and Intel GPU and CPU images.
Why it matters: Inference operators can route bounded scoring work through a dedicated endpoint and rebalance prefill and decode capacity without restarting the service. A new default cache implementation and a 779-pull-request release also create a large correctness,…
NVIDIA and Meta announced a multi-year expansion involving millions of Blackwell- and Rubin-generation GPUs, Grace CPUs, Spectrum-X networking and confidential-computing support for selected workloads including WhatsApp. The agreement framed infrastructure as a vertically integrated fleet spanning training, inference, networking and security.
Why it matters: Frontier competition increasingly depends on sustained power, networking, deployment software and supply commitments—not a one-time GPU purchase. Confidential-computing support also signals that privacy-sensitive consumer inference is becoming a hardware…
NVIDIA introduced the Rubin platform as a co-designed system spanning the Vera CPU, Rubin GPU, NVLink 6, ConnectX-9, BlueField-4 and Spectrum-6 Ethernet. NVIDIA said silicon was in full production, while partner systems were expected in the second half of 2026.
Why it matters: The competitive unit is becoming the rack-scale system—compute, networking, memory, CPU, security and software—not an isolated accelerator. That changes how buyers should compare infrastructure and why model-serving economics can improve without a…