Technology

real-time endpoints

1 synchronized story carry this tag.

SageMaker routes LLM requests using live KV-cache state

Amazon SageMaker AI real-time endpoints added KV-cache-aware routing that selects an instance holding the longest matching cached prompt prefix while accounting for load, then falls back to load-aware routing when no useful match is available.

Why it matters: Long prompts and multi-turn agents can avoid repeated prefill work without changing client APIs, but reported cache state, eviction, skew and fallback behavior become part of serving correctness and latency.

Read storyOriginal source · aws.amazon.com