Buoyant releases Kinference, an open source inference control plane for vLLM

Buoyant has released Kinference under the Apache 2.0 license. It is designed as an inference control plane for vLLM deployments on Kubernetes.

According to the announcement, Kinference adds declarative routing policies, KV-cache-aware load balancing and per-tenant metrics. Buoyant reports that in its own benchmark, median time to first token fell by 95%.

The available text does not describe the benchmark setup, hardware or workload, so the 95% figure rests on Buoyant's own testing.

Teams running vLLM on Kubernetes can set routing behavior through declarative policies, and KV-cache-aware balancing and per-tenant metrics are aimed at shared serving clusters. Because the latency gain comes from Buoyant's benchmark, treat it as a starting estimate and measure it on your own traffic before relying on it.