Our take
- Activity
- 71
- Community
- 49
- Issues
- 47
- Pull requests
- 72
+4 stars in the last 30 days
- Releases more often than 83% of the projects we track
- Slower to close issues than 81% of the projects we track
Project history
Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.
Stars
170Forks
29Change since Nov 17, 2026: +1 stars·+1 forks
Overview
agentevals evaluates agent behavior from recorded OpenTelemetry traces. It lets teams inspect and score what an agent actually did, such as which tools it called and what it answered, without rerunning the agent or spending tokens on another execution.
It is most useful for framework-agnostic, instrumented agents where behavior can be checked against expected tool trajectories, response criteria, or custom scoring rules. Evaluations can run offline from trace files, or through a local web interface and OTLP receiver. The project is under active development and notes that breaking changes are possible.
Key Features
- Reads Jaeger JSON and OTLP trace formats from OpenTelemetry-instrumented agents.
- Scores traces against golden evaluation sets, including tool-trajectory and response-match checks.
- Runs multiple evaluators and supports configurable trajectory matching.
- Supports custom evaluators in Python, JavaScript, or other languages, plus LLM-based judges.
- Provides an offline CLI for scripting and CI quality gates.
- Includes a web UI for trace inspection and live streaming through an OTLP receiver.
- Offers an MCP server for evaluating traces and inspecting live sessions.
- Can run locally without a required cloud service or database for basic evaluation.
Use Cases
- Agent developers can compare a recorded run with expected tool calls and responses while iterating on an instrumented agent.
- Platform and QA teams can add trace-based pass/fail checks to CI before deploying agent changes.
- Teams using multiple agent frameworks can reuse evaluation sets when each framework exports compatible OpenTelemetry traces.
- Operators debugging live behavior can stream traces to the UI and inspect tool calls and outputs as sessions run.
Getting Started
Install the CLI, REST API, and embedded web UI from PyPI:
pip install agentevals-cli
For usage examples and optional integrations, see the repository README.
Alternatives
- opik: Opik is a broader tracing and evaluation platform, while agentevals focuses on scoring existing OpenTelemetry traces locally without rerunning agents.
- phoenix: Phoenix combines trace inspection with evaluation and troubleshooting in a self-hosted platform; agentevals focuses on local evaluation of existing traces.
- weave: Weave organizes and evaluates runs within an experiment-tracking workflow, while agentevals scores existing OpenTelemetry traces without extra model calls.
- langwatch: LangWatch adds simulation, prompt management, and production monitoring, while agentevals focuses on evaluating existing traces locally.
| Project | Language | License | Stars | Status |
|---|---|---|---|---|
| agentevals | Python | Apache-2.0 | 162 | Not checked yet |
| opik | Python | Apache-2.0 | 22.8k | Active |
| phoenix | Python | Other | 11.7k | Active |
| weave | Python | Apache-2.0 | 1.1k | Active |
| langwatch | TypeScript | Apache-2.0 | 4.9k | Active |
Considerations
- Evaluation depends on traces being available in supported Jaeger JSON or OTLP formats; frameworks need compatible OpenTelemetry instrumentation.
- Results reflect recorded behavior. The project is designed to score existing traces rather than execute agents or replay test cases.
- The README describes the project as under active development and warns that breaking changes may occur.
- Durable evaluation run history requires enabling the preview Postgres backend. Basic CLI evaluation and default in-memory server operation do not require a database.
Found this useful?
Share it with someone who would like agentevals.