AI Agent News/stories/vercel-ai-gateway-now-supports-private-inference-for-self-hosted-and-fine-tuned.md

Infrastructure

Vercel AI Gateway now supports Private Inference for self-hosted and fine-tuned models

Vercel's AI Gateway can now route requests to private model endpoints, such as fine-tuned or self-hosted models, through the same authentication, SDKs, APIs and observability used for catalog models. Teams add an endpoint as a private provider and then call its models with a private/ model ID.

Source: vercel.com/changelog/ai-gateway-now-supports-private-inference

Enterprises running fine-tuned, self-hosted or prerelease models previously needed separate keys, client code and logs for each of them. Private Inference connects those model endpoints to AI Gateway so they can be managed alongside catalog models in one place.

Private models can be called through Chat Completions, Responses or Messages, and AI Gateway translates each request to the endpoint's format, so switching between private and catalog models only changes the model ID. Only the owning Vercel team can view or call private models, and they are not listed in the public catalog. Credentials are stored encrypted, and Zero Data Retention can be enabled per provider.

Setup can be done on the Private Inference page or with the Vercel CLI, version 65.16.0 or later, using the vercel ai-gateway private-inference commands. Private model requests appear in AI Gateway logs next to catalog traffic, and developers can configure a catalog model as a fallback if the private endpoint fails.

Why it matters if you build agents

Agents that rely on a fine-tuned or self-hosted model can now use the same AI Gateway keys, SDK calls and logging as their other models, without separate client code.

Switching an agent between a private model and a catalog model is a change to the model ID, and a catalog fallback can be set for when the private endpoint fails.

Developers should check the Private Inference documentation for supported model types, requirements and billing before relying on it in production.

Get stories like this every weekday

One short email with the day's agent tooling news.

Subscribe