SageMaker routes LLM requests using live KV-cache state
Amazon SageMaker AI real-time endpoints added KV-cache-aware routing that selects an instance holding the longest matching cached prompt prefix while accounting for load, then falls back to load-aware routing when no useful match is available.
Why it matters: Long prompts and multi-turn agents can avoid repeated prefill work without changing client APIs, but reported cache state, eviction, skew and fallback behavior become part of serving correctness and latency.