AWS adds bi-directional KV transfer to vLLM to lower TTFT in multi-turn agents

In prefill-decode disaggregated serving, KV cache normally moves only from the prefill node to the decode node. In multi-turn agent workloads, the decode node keeps KV for the previous response, but the prefill node cannot retrieve it, so it recomputes that context on every turn. The bi-directional mode lets the decode node return retained KV blocks to the prefill node on later turns.

The approach depends on a KV-cache-aware router that maps a conversation_id to the decode node's transfer parameters and stores only block-location metadata. Decode nodes hold retained blocks for a configurable lifetime, decoder_kv_blocks_ttl, which defaults to 480 seconds. A kv_recompute_threshold, defaulting to 64 tokens, decides whether pulling KV from the decode node beats recomputing it locally.

The benchmarks disabled prefix caching on the prefill node to mimic cache eviction. With the feature on, Qwen3-32B TTFT rose from 122 ms at 2k tokens to 460 ms at 24k, compared with 158 ms to 778 ms with it off. AWS says the feature works with vLLM v0.21.0 or later and NixlConnector, and results vary with hardware, model and workload.

Teams running self-hosted vLLM with prefill-decode disaggregation can cut repeated prefill work as conversation history grows, which keeps first-token latency steadier in long agent sessions.

The gains require a KV-cache-aware router and a retention TTL tuned to the typical gap between turns. Turns that arrive after the TTL expires fall back to normal prefill, so the benefit depends on session timing.