Agent-EvalKit is an open-source AI agent evaluation toolkit AWS released on June 11, 2026, licensed under Apache 2.0 (source: AWS Machine Learning Blog). It's designed for agents that already make tool calls, retrieve data, hit external APIs, or follow multi-step plans—type /evalkit.* commands in Claude Code, Kiro CLI, or Kilo Code, and the assistant reads your code, generates test cases, instruments traces, runs evaluations, and points you to the exact code that needs fixing.
Don't treat it as a replacement for "ask a few questions and eyeball the answers." It catches the cases where the final answer looks right but the tool was wrong, empty results were hallucinated into facts, or citations don't match the source.
When to Use Agent-EvalKit
If your agent meets any two of the following criteria, it's worth bringing Agent-EvalKit into your dev cycle instead of relying on manual spot checks:
| Condition | What to Evaluate |
|---|---|
| Calls search, databases, CRM, tickets, or internal APIs | Whether the right tool was selected, parameters are correct, and it stops on failure |
| Writes tool results into the final response | Whether the response is faithful to what the tools returned |
| Handles multi-turn tasks | Whether intermediate state, context inheritance, and tool order are stable |
| Planning to switch models, change system prompts, or update tool descriptions | Whether the same traces are comparable before and after the change |
| Your team has already spotted "sounds true but sources are empty" issues | Whether empty tool results are disclosed rather than filled in by the model |
If your agent is still pure chat—no external tools, no business actions—a manual checklist and a fixed prompt set are enough. Agent-EvalKit's value shows up when you need to see the execution path.
How It Differs from AgentCore Evaluation Datasets
We already have a guide on building AI agent evaluation datasets, which covers turning production failures into versioned datasets and release gates. Agent-EvalKit sits earlier in the pipeline: it analyzes code in your local dev directory, generates eval/ artifacts, runs traces, and produces reports.
| Question | Better with Agent-EvalKit | Better with AgentCore Dataset/Prod Evaluation |
|---|---|---|
| I just wrote an agent and want to know where it hallucinates | Yes | No |
| I want to inspect every tool call and intermediate state | Yes | Can supplement later |
| I want to freeze production failures as release gates | Can help | Yes |
| I need long-term production quality monitoring | No | Yes |
| I want the coding assistant to point out exactly which code to fix | Yes | No |
In practice, use Agent-EvalKit first to identify high-risk behaviors, then write consistently reproducible failures into a versioned evaluation dataset.
Prerequisites
| Item | Requirement |
|---|---|
| AWS account | Enable the foundation model you need for evaluation in the Amazon Bedrock console |
| Local environment | Python 3.11+, Git, uv |
| Coding assistant | Claude Code, Kiro CLI, or Kilo Code installed and runnable in your project |
| Agent code | Source code, tool definitions, prompts, and runtime configuration |
| Supported frameworks | AWS mentions Strands, LangGraph, and CrewAI for auto-instrumentation; check the Agent-EvalKit repo for the latest support matrix |
Install uv on macOS/Linux:
curl -LsSf https://astral.sh/uv/install.sh | sh
Install Agent-EvalKit:
uv tool install evalkit --from git+https://github.com/awslabs/Agent-EvalKit.git
Minimal Getting-Started Flow
Create an evaluation project and copy your agent code into it. The paths below are examples, not actual runs from this site:
evalkit init my-agent-evaluation
cd my-agent-evaluation
cp -r /path/to/your/agent .
If you're using Claude Code, start it from the evaluation directory:
claude
For a guided first run, use the quick command:
/evalkit.quick Evaluate my agent at ./my_agent for faithfulness, tool parameter accuracy, and response quality
For more control, run each phase individually:
/evalkit.plan Evaluate my agent at ./my_agent for faithfulness, tool parameter accuracy, and response quality
/evalkit.data
/evalkit.trace
/evalkit.run_agent
/evalkit.eval
/evalkit.report
These six phases generate an evaluation plan, test data, traces, scores, and a report in the eval/ directory. You can re-run individual phases later—for example, re-run /evalkit.eval after tweaking metrics, or /evalkit.run_agent and /evalkit.report after changing tool code.
What Each Phase Checks
| Phase | Command | Artifact | What to Check |
|---|---|---|---|
| Plan | /evalkit.plan |
Evaluation plan and metrics | Whether the metrics cover your real risks, not just how good the answers look |
| Data | /evalkit.data |
Test cases | Whether edge cases are covered: empty results, wrong parameters, insufficient permissions, stale data |
| Trace | /evalkit.trace |
OpenTelemetry-compatible tracing instrumentation | Whether tool calls, model responses, and intermediate state are captured |
| Run agent | /evalkit.run_agent |
Structured trace per test case | Whether the agent completes each test case as expected |
| Eval | /evalkit.eval |
Executable evaluation code and scores | Whether rule-based eval and LLM-as-judge each serve their role |
| Report | /evalkit.report |
Fix recommendations | Whether recommendations point to specific files, behaviors, and expected impact |
Don't skip the human review of the Data phase. The assistant can generate test cases, but real user behavior, business red lines, and historical incidents are yours to add.
Choosing the Three Core Metrics
AWS's example travel research agent used three metrics: Faithfulness, Tool Parameter Accuracy, and Response Quality. These work well for most tool-using agents:
| Metric | What It Detects | Best Evaluated By |
|---|---|---|
| Faithfulness | Whether the final response is grounded in tool data, or empty results were fabricated | Trace + LLM judge + citation check |
| Tool Parameter Accuracy | Whether the right tool was called with the right parameters | Rule check first; manual review when needed |
| Response Quality | Whether the answer is clear, complete, and actionable | LLM judge or manual sampling |
If your agent handles permissions, payments, deletions, or message sending, add hard rules:
- Don't call business APIs without passing permission checks
- When tools return empty results, the response must explicitly say "no data returned"
- High-risk actions must wait for human confirmation
Reading the Report
In AWS's example, the travel research agent scored 83.9% on Response Quality, 64.5% on Tool Parameter Accuracy, and only 32.3% on Faithfulness. It wasn't that the agent couldn't write travel advice—when the web search tool returned empty or incomplete results, it fabricated exchange rates, temperatures, and attraction details, presenting them as if they came from the tools.
Process reports in this order:
- Fix high-risk hallucinations first: empty tool results, wrong user identity, permission failures, stale real-time data.
- Fix tool parameter issues next: dates, regions, user IDs, currencies, product SKUs, filter conditions.
- Polish response quality last: formatting, summary order, wording.
Don't let a high Response Quality score greenlight a release. For tool-using agents, a polished answer can mask broken paths.
CI/CD Integration
Don't aim for full automation from day one. Start with local evaluation to establish a stable baseline, then wire the most critical scenarios into CI:
| Step | Pass Criteria |
|---|---|
| Lock in 20–50 high-risk test cases | Cover empty tool results, permission failures, parameter errors, stale data |
| Save traces and reports | Enable comparison across model or prompt changes |
| Set hard gates | Block on any high-risk faithfulness or permission failure |
| Set soft gates | Require human approval when response quality drops below a threshold |
| Always compare against the same test set | Don't swap test cases and compare scores at the same time |
Once the test set is stable, wire production traces into Amazon Bedrock AgentCore Observability and AgentCore Evaluation. AWS's official post also positions production monitoring as a follow-up, not a replacement for dev-time evaluation.
Common Mistakes
Only testing the final answer. The agent may get the right conclusion while skipping permission checks or using the wrong tool.
Treating LLM-as-judge as the sole arbiter. Semantic quality is fine for the judge. Permissions, tool order, parameters, and forbidden actions should be rule-checked first.
Skipping test case review. /evalkit.data can generate broad coverage, but your real-world business failures need to be added manually.
Loading up too many metrics at once. Start with faithfulness, tool parameter accuracy, and response quality. Add cost, latency, style, and safety metrics after the baseline is stable.
Only evaluating before release. Run evaluation after every meaningful prompt change, tool description update, model version switch, or retrieval strategy change. Otherwise, the report is just explaining problems that already happened.
FAQ
Does Agent-EvalKit require Amazon Bedrock?
AWS's post states that running evaluations requires an AWS account with a foundation model enabled in the Amazon Bedrock console for LLM-as-judge scoring. So if you're following the official workflow for a full evaluation, yes, you'll need Bedrock access.
Which coding assistants does it support?
Agent-EvalKit integrates with Claude Code, Kiro CLI, and Kilo Code. The official post uses Claude Code for its examples.
Does it auto-fix code?
The final phase /evalkit.report produces prioritized fix recommendations with code locations and expected impact. Whether to apply the changes is still up to you—the report is not an auto-fix tool.
I already have an evaluation dataset. Do I still need Agent-EvalKit?
If your dataset only stores inputs and expected outputs, Agent-EvalKit still adds value—it fills in traces, tool parameters, faithfulness checks, and code-level recommendations. If your evaluation platform already captures complete tool traces and produces actionable fix recommendations, the incremental value is smaller.
How should I write the prompt for the first run?
Write with specific risks, not generic "evaluate quality." For example:
/evalkit.plan Evaluate ./my_agent for hallucinations when tools return empty results, wrong tool parameters for date and region filters, and response quality for business users.
This focuses the evaluation on empty results, date/region parameters, and business usability, and the subsequent Data, Eval, and Report phases will follow the same direction.