AWS releases UAEF for cross-framework AI agent evaluation

AWS has released the Universal Agent Evaluation Framework, an open-source system for evaluating AI agents built with different frameworks through a shared trace format and scoring layer. It is designed to let teams compare changes to prompts, models or tools without rebuilding an evaluation pipeline around each framework’s native records.

UAEF adapters read messages, tool calls and agent-to-agent handoffs from a run, then convert them into a canonical AgentTrace. That common record feeds the rest of the system, keeping trace collection separate from scoring.

The public UAEF repository lists adapters for LangGraph, LangChain, Amazon Bedrock Agents, Langfuse, Strands and Amazon Bedrock AgentCore, along with a GenericJSON adapter for custom frameworks. Teams can normalize existing traces directly. UAEF can also invoke an agent through a runtime ARN or HTTP endpoint, collect the response and adapt the result.

The framework includes 36 built-in metrics across seven dimensions: tool calling, response quality, responsible AI, performance, multi-turn behavior, multi-agent behavior and reasoning. AWS says each metric returns a normalized score from 0 to 1, with 1 best. That puts dimension scores and the overall result on the same scale.

Its metric set combines deterministic calculations with LLM-as-a-judge assessments run through Amazon Bedrock. UAEF also supports custom metrics and integrations with RAGAS and DeepEval.

Each evaluation takes an agent and a dataset. Reference answers are optional, although tool-calling tests need expected calls that specify the tools, their order and their arguments.

With persistence enabled, UAEF stores experiment metadata and average scores in Amazon DynamoDB, while full evaluation results are written as JSON to Amazon S3. Teams can compare stored runs with a baseline on the metrics they share. The comparison engine calculates absolute and percentage changes, then labels a result as an improvement, regression or unchanged using a configurable threshold that defaults to 5%.

AWS defines regression bands of 5% to 10% as minor, 10% to 20% as moderate and more than 20% as severe, grouped by evaluation dimension. The Python library and deployable web interface share the same evaluation engine and storage layer.

AWS also says UAEF can screen canonical traces for personally identifiable information before storage and redact the saved copy. The project is available under the Apache 2.0 license.

According to AWS, customers across 12 organizations in telecommunications, healthcare and financial services used UAEF to evaluate agents and compare versions before release. The company did not identify those organizations or publish independent benchmark results showing equivalent scores across adapters.