agentevals vs promptfoo
Agent trace evaluation compared with LLM app testing
agentevals evaluates recorded OpenTelemetry traces to check how an agent behaved, while promptfoo evaluates prompts and models and red-teams LLM applications. Their main difference is the evaluation target: existing agent runs for agentevals, versus prompt and application testing for promptfoo.
agentevals: Evaluate AI Agents from OpenTelemetry Traces
agentevals scores AI agent behavior from existing OpenTelemetry traces, without rerunning agents or making extra model calls. It suits teams building instrumented agents that need local evaluation, golden-set checks, or CI quality gates.
promptfoo: Evaluate and Red-Team LLM Apps
Promptfoo is a TypeScript CLI and library for evaluating and red-teaming LLM-based apps. Compare prompts and models, scan for vulnerabilities, and run checks in CI/CD, with evaluations performed locally.
| agentevals | promptfoo | |
|---|---|---|
| Language | Python | TypeScript |
| License | Apache-2.0 | MIT |
| Stars | 162 | 26.4k |
| Forks | 28 | 2.6k |
| Last analyzed | Oct 4, 2026 | Oct 24, 2026 |
Key differences
- agentevals scores existing Jaeger JSON or OTLP traces; promptfoo compares prompt and model outputs across supported providers.
- agentevals focuses on tool trajectories, responses, and custom trace scoring; promptfoo also includes red-team tests and vulnerability scans.
- agentevals is written in Python and licensed under Apache-2.0; promptfoo is written in TypeScript and licensed under MIT.
- agentevals supports offline CLI evaluation and a local UI with an OTLP receiver; promptfoo offers a CLI and Node.js package, with evaluations described as local.
- agentevals requires compatible OpenTelemetry instrumentation and available traces; promptfoo evaluations typically require API keys for the providers being tested.
- agentevals is under active development and warns of possible breaking changes; promptfoo is also actively developed, with frequent commits and releases.
Choose agentevals if you…
- need to score recorded agent behavior without rerunning agents or making extra model calls.
- want to check tool trajectories or responses from OpenTelemetry traces in local workflows or CI.
- work with instrumented agents across frameworks that export compatible traces.
Choose promptfoo if you…
- want to compare prompts or candidate models using consistent evaluation cases.
- need to red-team an LLM application or scan it for vulnerabilities.
- want to integrate LLM evaluation and security checks into CI/CD using a CLI or Node.js package.
Compare the numbers side by side, or add more projects
This comparison is generated with AI from the OSRepos analyses of both projects. Always check each project's repository and documentation before choosing.