agent-anvil vs agentevals
Agent workflow testing and trajectory evaluation compared
agent-anvil and agentevals both evaluate AI agent behavior beyond final answers, including tool use and execution steps. agent-anvil centers on YAML scenarios, trace checks, and CI workflows, while agentevals provides Python and TypeScript trajectory evaluators, including graph-based options.
agent-anvil: Test AI Agent Tool Use in CI
Agent Anvil evaluates tool-using AI agents through scenario-based runs, trace checks, and optional semantic grading. It helps teams catch unsafe or incorrect tool behavior before it reaches production.
agentevals: Evaluate AI Agent Execution Trajectories
AgentEvals provides Python and TypeScript evaluators for checking the steps AI agents take, including tool calls and graph paths. Use it to compare runs with references or have an LLM judge trajectory quality.
| agent-anvil | agentevals | |
|---|---|---|
| Language | Python | Python |
| License | MIT | MIT |
| Stars | 9 | 737 |
| Forks | 2 | 60 |
| Last analyzed | Oct 4, 2026 | Oct 3, 2026 |
Key differences
- agent-anvil runs repeatable YAML scenarios and checks trace invariants, policies, and optional semantic grading; agentevals compares trajectories with references or uses an LLM judge.
- agent-anvil includes reports, failure clusters, repair suggestions, and a provided GitHub Action; agentevals integrates with LangSmith and testing workflows such as pytest and Vitest/Jest.
- agent-anvil supports demo agents and external agents through command or HTTP adapters; agentevals offers graph trajectory evaluation and LangGraph thread utilities.
- agent-anvil is described as a Python harness and specifies Python 3.12 or newer; agentevals provides Python and TypeScript evaluators and supports asynchronous evaluation in Python.
- Both projects use the MIT license. In the supplied facts, agentevals has 737 stars and 60 forks, while agent-anvil has 9 stars and 2 forks.
Choose agent-anvil if you…
- need repeatable YAML scenarios and checks for tool order, arguments, call limits, or workflow policies.
- want CI reports and failure investigation aids, including trace artifacts and suggested repair plans.
- need to evaluate an existing command-line or HTTP agent without replacing its runtime.
Choose agentevals if you…
- want Python or TypeScript evaluators for comparing agent execution trajectories.
- need flexible trajectory matching, custom tool-argument comparators, or graph-path checks for LangGraph workflows.
- want an LLM judge for trajectory quality or integration with LangSmith evaluation and test workflows.
Compare the numbers side by side, or add more projects
This comparison is generated with AI from the OSRepos analyses of both projects. Always check each project's repository and documentation before choosing.