Our take
- Activity
- 42
- Community
- 47
- Issues
- 21
- Pull requests
- 43
+8 stars in the last 30 days
Project history
Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.
Stars
759Forks
66Change since Nov 15, 2026: 0 stars·0 forks
Overview
AgentEvals is a Python and TypeScript library for evaluating an AI agent’s execution path, not just its final answer. It helps teams test whether an agent uses appropriate tools and follows acceptable steps, even when multiple paths could produce a useful result.
It fits projects that need trajectory-focused checks alongside broader evaluation tools. Evaluators can compare runs with expected trajectories or use an LLM judge, and graph-based options support LangGraph workflows.
Key Features
- Compare message trajectories in strict, unordered, subset, or superset modes.
- Configure tool-call argument matching, including custom per-tool comparators.
- Use LLM-as-judge evaluators with customizable prompts and scoring options.
- Evaluate graph trajectories by visited nodes, with utilities to extract trajectories from LangGraph threads.
- Check graph steps for an exact match against a reference.
- Run evaluators asynchronously in Python.
- Integrate evaluation runs with LangSmith and its pytest, Vitest/Jest, and
evaluateworkflows.
Use Cases
- Agent developers can catch changes in tool choice or call order during regression testing.
- Teams with multiple valid execution paths can allow reordered or additional tool calls while still checking required behavior.
- LangGraph developers can evaluate node paths across multi-turn threads and human-in-the-loop interruptions.
- Evaluation teams can use an LLM judge when there is no single exact reference trajectory.
Getting Started
Install the Python package:
pip install agentevals
For TypeScript, install with npm install agentevals @langchain/core. LLM-as-judge examples require an LLM provider and credentials. See the README for setup and usage details.
Alternatives
- Agent Anvil: Agent Anvil runs CI scenario suites for tool-using agents, capturing traces and grading failures, while AgentEvals focuses on reusable trajectory evaluators.
- giskard-oss: Giskard emphasizes repeatable agent and RAG checks, vulnerability probing, and adversarial safety tests rather than focused trajectory comparison.
- Promptfoo: Promptfoo evaluates and red-teams prompts, agents, and RAG apps through a CLI and declarative configs, while AgentEvals centers on trajectory evaluators.
| Project | Language | License | Stars | Status |
|---|---|---|---|---|
| agentevals | Python | MIT | 737 | Not checked yet |
| agent-anvil | Python | MIT | 9 | New |
| giskard-oss | Python | Apache-2.0 | 5.9k | Active |
| promptfoo | TypeScript | MIT | 26.4k | Active |
Considerations
- LLM-as-judge evaluation requires an LLM client and may require provider credentials; results depend on the chosen model and prompt.
- Reference-based match evaluators require a suitable reference trajectory, and strict matching may reject runs whose tool calls or ordering differ.
- This package focuses on agent trajectories. The README points to OpenEvals for more general evaluation tools.
Found this useful?
Share it with someone who would like agentevals.