TensorCast opens a shared tensor-state layer for vLLM and SGLang
TensorCast manages weights, KV caches, and checkpoints independently of serving workers. Under its heaviest medium-delay workload, its router cut median time-to-first-token by 93.2% versus a load-aware router using Mooncake.
Why it matters: Moving and duplicating persistent tensor state is becoming a separate scaling bottleneck from model computation.