Red Hat, IBM and Google are lead maintainers of llm-d, which they co-founded last year with CoreWeave and NVIDIA. AMD, Cisco, Hugging Face, Intel, Lambda and Mistral AI have joined as contributors. The project lets teams deploy inference stacks for any model, on any accelerator and in any cloud.
Version 1.0 focuses on day 2 operations, meaning the work of running an AI service after it is deployed. Its flow control and prioritization are meant to keep bursty workloads, such as agents, from overwhelming latency-sensitive traffic. Requests can also stay pinned to the same infrastructure across turns so their prefix caches stay warm.
In a test of an agentic coding workload, llm-d's prefix-aware router captured 97% of the available cache reuse. IBM and Red Hat also reported running 3,000 software-engineering agents concurrently on a self-hosted deployment, at 5 to 10 times less than the price of serving the same open model as a service.
Why it matters if you build agents
If you run agents on shared inference, llm-d 1.0 is aimed at letting several tenants and priority levels share one endpoint with less manual tuning. Agents that repeatedly reread the same context may benefit from cache-aware routing, which can lower serving costs. Check how the stable API and upgrade behavior fit your existing Kubernetes operations before adopting it.
Get stories like this every weekday
One short email with the day's agent tooling news.