# IBM, Red Hat and Google release llm-d 1.0 inference framework with multi-tenant controls

published: 2026-10-20T13:05:27.428Z
topic: infrastructure
source: https://siliconangle.com/2026/10/20/ibm-backed-llm-d-framework-adds-controls-for-running-ai-models-in-house/
url: https://aiagentnews.io/stories/ibm-red-hat-and-google-release-llm-d-1-0-inference-framework-with-multi-tenant

IBM, Red Hat and Google have released version 1.0 of the llm-d inference framework, adding controls for running models on enterprise infrastructure. The update targets agent traffic with multi-tenant prioritization, zero-downtime upgrades and cache-aware routing.

IBM, Red Hat and Google, the lead maintainers of the open-source llm-d project, have released version 1.0 of its distributed inference framework for Kubernetes. The release adds operational controls for running large models on an organization's own infrastructure, including quality of service across multiple tenants, zero-downtime operations and serving patterns for AI agents. Flow controls and prioritization are meant to stop bursts of agent activity from overwhelming services that need fast responses, such as coding assistants.

Version 1.0 routes requests from agents that make several model calls to replicas already holding relevant cached context, and it can move cached data between GPU memory, CPU memory and storage when GPU capacity runs short. IBM says the framework's router captured 97% of available cache reuse in a recent agentic coding test. In a separate demonstration, IBM and Red Hat ran 3,000 software-engineering agents concurrently on self-hosted infrastructure at a cost five to 10 times lower than a hosted service serving the same open model.

llm-d works with inference engines including vLLM and SGLang and is currently hosted as a sandbox project by the Cloud Native Computing Foundation. Other contributors include AMD, Cisco, Hugging Face, Intel and Mistral AI.

## Why it matters if you build agents

Teams running agents on their own clusters can now get routing, cache reuse and tenant prioritization from a 1.0 release instead of assembling them by hand. Agents that pause for tool calls and return can reuse earlier context, which IBM says reduces repeated computation. Check that your serving stack uses vLLM or SGLang before adopting it, since llm-d sits on top of those engines.