agentevals vs promptfoo
AI evaluation tools compared
agentevals evaluates the steps AI agents take, including tool calls and graph paths. promptfoo evaluates prompts, models, and LLM applications, with red teaming and vulnerability checks; the main difference is trajectory-focused testing versus broader application evaluation and security testing.
agentevals: Evaluate AI Agent Execution Trajectories
AgentEvals provides Python and TypeScript evaluators for checking the steps AI agents take, including tool calls and graph paths. Use it to compare runs with references or have an LLM judge trajectory quality.
promptfoo: Evaluate and Red-Team LLM Apps
Promptfoo is a TypeScript CLI and library for evaluating and red-teaming LLM-based apps. Compare prompts and models, scan for vulnerabilities, and run checks in CI/CD, with evaluations performed locally.
| agentevals | promptfoo | |
|---|---|---|
| Language | Python | TypeScript |
| License | MIT | MIT |
| Stars | 737 | 26.4k |
| Forks | 60 | 2.6k |
| Last analyzed | Oct 3, 2026 | Oct 24, 2026 |
Key differences
- agentevals is a Python and TypeScript library focused on agent execution trajectories; promptfoo is a TypeScript CLI and library for evaluating prompts, models, and LLM applications.
- agentevals compares trajectories with references or uses an LLM judge, while promptfoo compares model outputs across supported providers and runs configured evaluations.
- agentevals includes graph trajectory evaluation and LangGraph utilities; promptfoo includes red teaming, vulnerability scanning, and code scanning for LLM-related security and compliance issues.
- promptfoo describes CI/CD workflows and a web viewer for results; agentevals integrates with LangSmith evaluation and test workflows.
- Both projects use the MIT license. The supplied facts list 737 stars and 60 forks for agentevals, compared with 25.7k stars and 2.4k forks for promptfoo.
- agentevals notes that reference-based evaluators need suitable trajectories and that LLM judges depend on the model and prompt; promptfoo notes that most providers require an API key and results depend on providers and test cases.
Choose agentevals if you…
- need to check an agent's tool choices, call order, or other execution steps.
- want to evaluate graph paths in LangGraph workflows, including multi-turn threads.
- need trajectory comparisons that allow reordered or additional calls, or an LLM judge when no single reference fits.
Choose promptfoo if you…
- want to compare prompt revisions or model outputs across supported providers.
- need red-team testing or vulnerability scans for an LLM application.
- want to run evaluation or security checks locally or in CI/CD and review results in a web viewer.
Compare the numbers side by side, or add more projects
This comparison is generated with AI from the OSRepos analyses of both projects. Always check each project's repository and documentation before choosing.