Open Source Evaluation Tools
Evaluation is the systematic measurement of how well a model or AI application performs against defined goals. It helps teams compare systems, detect regressions, assess answer quality and safety, and understand where outputs fall short. Tests may use fixed datasets, simulated scenarios, human judgments, or automated metrics, depending on the task and the risks involved. For language models and agents, evaluation can also examine reasoning steps, tool use, retrieved information, and consistency across repeated runs.
Open source tools in this area include benchmark runners, dataset and test-case generators, scoring frameworks, experiment trackers, and systems for inspecting traces or monitoring results. When choosing one, consider its maturity, license, maintenance activity, supported models and integrations, data requirements, and whether its metrics fit your use case. These tools are useful to researchers, developers, and teams building or operating AI systems.
54 repositories · updated October 13, 2026
awesome-agent-harness: Curated Guide to Agent Harness Engineering
A bilingual, annotated directory of resources on building the systems around AI agents, from context and tools to memory, evaluation, and coding-agent design. Useful for product teams that want practical reading paths rather than an unannotated link dump.
NewHealth 30/100
skill-up: Evaluate and Improve Agent Skills
skill-up is a Go CLI for testing Agent Skills, agents, and workspaces with repeatable cases and structured reports. It suits teams that want to catch behavior regressions or iteratively improve Skills using evaluation results.
NewHealth 69/100
agentevals: Evaluate AI Agents from OpenTelemetry Traces
agentevals scores AI agent behavior from existing OpenTelemetry traces, without rerunning agents or making extra model calls. It suits teams building instrumented agents that need local evaluation, golden-set checks, or CI quality gates.
ActiveHealth 61/100
agent-anvil: Test AI Agent Tool Use in CI
Agent Anvil evaluates tool-using AI agents through scenario-based runs, trace checks, and optional semantic grading. It helps teams catch unsafe or incorrect tool behavior before it reaches production.
NewHealth 42/100
uni-agent: Train Long-Horizon Agents at Scale
Uni-Agent is a Python framework for running and training long-horizon agents through a shared interaction stack. It helps agent developers connect harnesses, tasks, tools, and sandboxes for scalable inference, evaluation, and reinforcement learning.
ActiveHealth 61/100
benchmark-radar: Discover AI Benchmarks and Track Evaluation Scores
Benchmark Radar gathers AI benchmark records and evaluation evidence from public sources in a searchable catalog. Use it to discover benchmarks, inspect reported scores and citations, and follow new findings through its dashboard or offline CLI.
NewHealth 63/100
maka: Run and Inspect AI Agent Workflows
Apache Maka is an agent workspace that records model messages, tool calls, and permission decisions as an append-only event log. It is aimed at people building or evaluating agents who want local session data and a recoverable execution history.
NewHealth 81/100
ctx-gate: Reduce LLM Context Tokens Through a Proxy
ctx-gate is a Python proxy that compresses LLM conversation context while aiming to preserve facts relevant to the current prompt. It supports Anthropic and OpenAI-compatible APIs, with evaluation tools to check retention and token savings.
NewHealth 20/100
Awesome-Self-Evolving-Agents: Explore Research on Evolving Agents
A curated research index and survey companion covering self-evolving AI agents. It organizes papers, benchmarks, libraries, and applications for researchers and practitioners exploring how agents improve through model, environment, and combined evolution.
ActiveHealth 50/100
Awesome-Self-Improving-Agents: Explore Self-Improving Agent Research
A curated research hub for self-improving foundation-model agents, organized around model and agent-scaffold improvements. Researchers and practitioners can use it to find papers, benchmarks, and related resources by topic.
ActiveHealth 45/100
maka: Run Traceable AI Agent Workflows
Apache Maka is a local-first workspace for running AI agents through desktop, terminal, and evaluation tools. Its append-only event log records model messages, tool calls, and permission decisions for review and recovery.
NewHealth 81/100
envharness: Adapt Environments for Agent Learning
EnvHarness wraps frozen interactive environments with composable layers that can target an AI agent’s weaknesses without changing benchmark code or grading. It is for researchers building and evaluating agent-training workflows across different environments.
NewHealth 27/100