giskard-oss vs agentevals
LLM agent testing and evaluation compared
giskard-oss and agentevals are Python tools for evaluating AI agents, including multi-turn behavior. giskard-oss covers scenarios, RAG quality, and adversarial security testing, while agentevals focuses on agent execution trajectories and tool or graph steps, with Python and TypeScript support.
giskard-oss: Test and Red-Team LLM Agents
Giskard is a Python toolkit for evaluating agent behavior and probing AI systems for vulnerabilities. It suits teams building LLM agents or RAG applications that need repeatable checks, safety testing, and adversarial evaluation.
agentevals: Evaluate AI Agent Execution Trajectories
AgentEvals provides Python and TypeScript evaluators for checking the steps AI agents take, including tool calls and graph paths. Use it to compare runs with references or have an LLM judge trajectory quality.
| giskard-oss | agentevals | |
|---|---|---|
| Language | Python | Python |
| License | Apache-2.0 | MIT |
| Stars | 5.9k | 737 |
| Forks | 547 | 60 |
| Last analyzed | Oct 3, 2026 | Oct 3, 2026 |
Key differences
- giskard-oss includes scenario-based checks for outputs and traces, RAG grounding and quality checks, and adversarial vulnerability scans; agentevals focuses on comparing execution trajectories and judging their quality.
- agentevals supports Python and TypeScript, while giskard-oss is a Python toolkit.
- giskard-oss uses an Apache-2.0 license; agentevals uses an MIT license.
- agentevals offers trajectory matching in strict, unordered, subset, and superset modes, plus graph-path evaluation; giskard-oss offers checks such as semantic similarity, groundedness, and LLM-as-judge assessments.
- giskard-oss v3 requires Python 3.12 or later, and its v2 release remains available for legacy tabular-model scanning; agentevals supports asynchronous evaluation in Python.
Choose giskard-oss if you…
- need scenario-based checks for multi-turn agents or RAG applications.
- want adversarial tests for issues such as prompt injection, harmful content, or misinformation.
- need output checks such as groundedness, semantic similarity, or LLM-as-judge assessments.
Choose agentevals if you…
- need to check agent tool calls, arguments, or execution order against references.
- want flexible trajectory matching when several execution paths may be acceptable.
- need graph-path evaluation for LangGraph workflows or Python and TypeScript evaluators.
Compare the numbers side by side, or add more projects
This comparison is generated with AI from the OSRepos analyses of both projects. Always check each project's repository and documentation before choosing.