AI Agent News/stories/amazon-sagemaker-ai-adds-kv-cache-aware-routing-for-real-time-endpoints.md

Infrastructure

Amazon SageMaker AI adds KV cache aware routing for real-time endpoints

Amazon SageMaker AI now offers KV_AWARE routing for real-time endpoints, which sends each request to an instance that holds a matching cached prompt prefix and is not too busy. It is aimed at LLM workloads with multi-turn conversations, long shared prompts, or heavy load, and works with the same invocation APIs as before.

Source: aws.amazon.com/blogs/machine-learning/amazon-sagemaker-ai-real-time-endpoints-now-support-kv-cache-aware-routing

SageMaker AI takes cache state reported by the model container, which must be a supported AWS vLLM Deep Learning Container with prefix caching and KV cache events turned on. The endpoint keeps a managed index of cached token content, mapped to the instance or inference component copy that holds it, and never stores prompts or KV tensors. Requests go to the instance with the longest cached match, weighed against how busy each candidate is, and fall back to load-aware routing when no useful match exists.

Developers enable it by adding the SM_VLLM_KV_EVENTS_CONFIG environment variable to the model and setting RoutingStrategy to KV_AWARE in the production variant's RoutingConfig. Routing is active only with at least two instances or two inference component copies. Requests that set a non-null cache_salt are rejected with a 400 error, and InvokeEndpointWithBidirectionalStream is not supported.

In AWS's benchmark with Llama 3.1 70B Instruct on seven ml.p5.48xlarge instances, KV-aware routing cut p50 time to first token by up to 97 percent compared with random routing under long prompts and load. In a smaller two-instance test with Qwen3-4B-Instruct-2507, model latency averaged about 1.2 to 1.3 seconds against 4.1 to 5.1 seconds for random routing, though AWS notes that results depend on the workload.

Why it matters if you build agents

Teams running multi-turn chat, RAG or agent workloads on SageMaker AI can keep conversations on the instance that already holds their context, which can reduce time to first token when cache is under memory pressure.

Prefix-aware routing remains the simpler option when prompts are short, load is moderate, or the container is not supported, and AWS recommends comparing both strategies on your own traffic.

Endpoints do not log KV-aware routing decisions, so developers should check cached_tokens per response and the KV cache hit rate to confirm the routing is working.

Get stories like this every weekday

One short email with the day's agent tooling news.

Subscribe