AWS proposes push-based KV transfer to cut TTFT in vLLM disaggregated inference

vLLM splits LLM inference into prefill and decode phases, which can run on separate instances. The default NIXL connector uses a pull model: after prefill, the router relays block IDs to the decode instance, which allocates destination blocks and then issues a NIXL READ. These steps run one after another before the first token can be generated, which lengthens time-to-first-token.

The new push mode has the decode instance register its destination blocks in advance, so the prefill instance can start a NIXL WRITE as soon as its KV blocks are ready. A dedicated background thread on each prefill worker prepares descriptors and submits the transfer, keeping that work out of the main execution loop. Requests are paired by ID, and the connector handles registration and prefill completing in either order.

In the 1P1D benchmark, which disabled prefix caching so every request transferred the full KV cache, pull mode reached about 1,744 ms TTFT at 8 queries per second with 16k inputs, while push mode stayed near 540 ms. TPOT for pull rose to about 10.4 ms in the same range, while push held near 7 ms. The authors note that results vary by workload and configuration, and the feature was built on Trainium and contributed to vLLM as an accelerator-agnostic feature.

If you run disaggregated prefill and decode with vLLM and can accept choosing the decode instance before prefill finishes, switching the configuration to NixlPushConnector may lower TTFT under load.

Pull mode remains the better choice when you need bidirectional KV transfer or want load-aware routing that picks the decode instance after prefill.

Benchmark gains were measured on one model and one hardware setup, so you should measure your own workload before relying on the reported numbers.