AI Agent News/stories/aws-publishes-sample-routing-coding-agent-requests-between-bedrock-models-with.md

Infrastructure

AWS publishes sample routing coding agent requests between Bedrock models with NVIDIA NeMo Switchyard

AWS has published a sample that deploys LiteLLM with NVIDIA's NeMo Switchyard routing plugin on AWS, so coding agent requests can be sent to either Claude Opus 5.5 or Nemotron 3 Super 120B on Amazon Bedrock. The post recommends judging routing by cost per solved task rather than by price per token.

Source: aws.amazon.com/blogs/machine-learning/route-coding-agent-requests-to-the-right-model-on-amazon-bedrock-with-nvidia-nemo-switchyard

The sample uses the AWS CDK to run LiteLLM and the Switchyard plugin in one container on Amazon ECS with AWS Fargate, in private subnets. Coding agents send OpenAI-compatible requests with the model field set to switchyard. Stage routing reads recent tool activity and results in the request history to choose between a capable model and a lower-cost one, while Random routing splits traffic by weight for A/B tests.

The load balancer forwards only the chat completions, models and health paths, and AWS WAF applies IP reputation and rate-limit rules. Access is limited to an IPv4 range that the operator sets, and the gateway key is generated in AWS Secrets Manager. Inference calls go through a Bedrock Runtime interface endpoint in the VPC. AWS labels the sample experimental and intended for non-production accounts.

The included benchmark runners execute coding tasks through the gateway and report cost per solved task, which is the total cost of runs divided by the runs whose tests pass. A second runner replays multi-turn conversations to compare full chats across model groups. The authors say the bundled tasks are small and that a few runs are not enough to draw conclusions about cost or quality.

Why it matters if you build agents

Developers can compare a routed model group with single-model baselines on their own code and see whether a cheaper model lowers the cost of finished tasks, not just the cost of each call. Stage needs the full conversation history on every request, and unsupported message shapes fail before inference. Clients that use the OpenAI Responses or Anthropic Messages paths get a 404 from this deployment until those paths are added.

Get stories like this every weekday

One short email with the day's agent tooling news.

Subscribe