# AWS releases UAEF, a framework-agnostic system for evaluating AI agents

published: 2026-11-11T16:05:38.937Z
topic: infrastructure
source: https://aws.amazon.com/blogs/machine-learning/uaef-a-framework-agnostic-system-for-closed-loop-evaluation-of-ai-agents/
url: https://aiagentnews.io/stories/aws-releases-uaef-a-framework-agnostic-system-for-evaluating-ai-agents

AWS has released UAEF, an open source evaluation system that scores agents built with different frameworks using one set of metrics. It converts traces from LangGraph, LangChain, Strands Agents and Amazon Bedrock AgentCore into a common format before scoring them.

UAEF uses adapters to turn each framework's run records, including messages, tool calls and agent-to-agent handoffs, into one canonical trace. It then screens that trace for personally identifiable information such as email addresses, phone numbers and credit card numbers before storing results. The evaluation engine scores each trace with 36 built-in metrics across seven dimensions, including tool calling, response quality, responsible AI and multi-turn conversation.

Each evaluation can be saved as an experiment, with run metadata in Amazon DynamoDB and full results in Amazon S3. A team can mark one run as the baseline and compare later agent versions against it, with regressions flagged using a default threshold of 5 percent. Generated reports and LLM-based insights point to what to change in the prompt, model or tools before the next evaluation.

UAEF can be used as a Python library in notebooks and CI/CD pipelines, or as a deployable AWS service with a web interface. Both paths run the same evaluation engine and write to the same stores. Customers across 12 organizations in telecom, healthcare and financial services used it to compare agent versions before release.

## Why it matters if you build agents

Developers can keep one set of metrics when they switch frameworks or run agents on more than one, instead of rebuilding scoring logic for each framework. Comparing against a stored baseline shows whether a prompt, model or tool change actually improved the agent before release.

Judge-based metrics are evaluation signal and should not gate a security decision. Runtime enforcement on the agent still needs Amazon Bedrock Guardrails in the agent's own invocation path.