What changed
Amazon SageMaker AI real-time endpoints added KV-cache-aware routing that selects an instance holding the longest matching cached prompt prefix while accounting for load, then falls back to load-aware routing when no useful match is available.
Why it matters
Long prompts and multi-turn agents can avoid repeated prefill work without changing client APIs, but reported cache state, eviction, skew and fallback behavior become part of serving correctness and latency.
What to do
Replay a fixed long-context trace against random, prefix-aware and KV-aware routing; measure first-token latency, hit rate, load balance, eviction behavior and fallback during instance loss.
Caveat
The AWS launch post is dated 17 November, but the product availability notice establishes 16 November. It requires a compatible AWS vLLM container and replicated capacity.