langchain-ai

agentevals: Evaluate AI Agent Execution Trajectories

AgentEvals provides Python and TypeScript evaluators for checking the steps AI agents take, including tool calls and graph paths. Use it to compare runs with references or have an LLM judge trajectory quality.

View on GitHub Compare
Stars
737
Forks
60
Stars, last 30 days
+8

Our take

Health score

38/100

How it is scored
Activity
42
Community
47
Issues
21
Pull requests
43

+8 stars in the last 30 days

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

759

Forks

66

Change since Nov 15, 2026: 0 stars·0 forks

Overview

AgentEvals is a Python and TypeScript library for evaluating an AI agent’s execution path, not just its final answer. It helps teams test whether an agent uses appropriate tools and follows acceptable steps, even when multiple paths could produce a useful result.

It fits projects that need trajectory-focused checks alongside broader evaluation tools. Evaluators can compare runs with expected trajectories or use an LLM judge, and graph-based options support LangGraph workflows.

Key Features

  • Compare message trajectories in strict, unordered, subset, or superset modes.
  • Configure tool-call argument matching, including custom per-tool comparators.
  • Use LLM-as-judge evaluators with customizable prompts and scoring options.
  • Evaluate graph trajectories by visited nodes, with utilities to extract trajectories from LangGraph threads.
  • Check graph steps for an exact match against a reference.
  • Run evaluators asynchronously in Python.
  • Integrate evaluation runs with LangSmith and its pytest, Vitest/Jest, and evaluate workflows.

Use Cases

  • Agent developers can catch changes in tool choice or call order during regression testing.
  • Teams with multiple valid execution paths can allow reordered or additional tool calls while still checking required behavior.
  • LangGraph developers can evaluate node paths across multi-turn threads and human-in-the-loop interruptions.
  • Evaluation teams can use an LLM judge when there is no single exact reference trajectory.

Getting Started

Install the Python package:

pip install agentevals

For TypeScript, install with npm install agentevals @langchain/core. LLM-as-judge examples require an LLM provider and credentials. See the README for setup and usage details.

Alternatives

  • Agent Anvil: Agent Anvil runs CI scenario suites for tool-using agents, capturing traces and grading failures, while AgentEvals focuses on reusable trajectory evaluators.
  • giskard-oss: Giskard emphasizes repeatable agent and RAG checks, vulnerability probing, and adversarial safety tests rather than focused trajectory comparison.
  • Promptfoo: Promptfoo evaluates and red-teams prompts, agents, and RAG apps through a CLI and declarative configs, while AgentEvals centers on trajectory evaluators.
ProjectLanguageLicenseStarsStatus
agentevalsPythonMIT737Not checked yet
agent-anvilPythonMIT9New
giskard-ossPythonApache-2.05.9kActive
promptfooTypeScriptMIT26.4kActive

Considerations

  • LLM-as-judge evaluation requires an LLM client and may require provider credentials; results depend on the chosen model and prompt.
  • Reference-based match evaluators require a suitable reference trajectory, and strict matching may reject runs whose tool calls or ordering differ.
  • This package focuses on agent trajectories. The README points to OpenEvals for more general evaluation tools.

Found this useful?

Share it with someone who would like agentevals.

Comparisons