Our take
- Activity
- 86
- Community
- 82
- Issues
- 78
- Pull requests
- 61
+46 stars in the last 30 days
- Releases more often than 88% of the projects we track
- Merges more pull requests than 88% of the projects we track
Project history
Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.
Stars
5,953Forks
573Change since Nov 15, 2026: +2 stars·+2 forks
Overview
Giskard is a modular Python toolkit for testing and evaluating LLM-based agents, including multi-turn systems and RAG applications. It addresses the difficulty of validating systems whose outputs can vary, combining scenario-based checks with automated vulnerability scanning.
Use it to define evaluations for a callable system, run checks against its outputs or traces, and generate adversarial test suites. The current v3 packages focus on agent evaluation and security; legacy automatic tabular-model scanning remains in v2.
Key Features
- Define scenarios and suites for synchronous or asynchronous targets.
- Run checks such as string matching, regex, semantic similarity, groundedness, and LLM-as-judge assessments.
- Evaluate multi-turn interactions, not only individual prompts and responses.
- Scan agents with generated adversarial tests for prompt injection, harmful content, stereotypes, and misinformation.
- Evaluate RAG answer quality and grounding.
- Use modular packages for checks, scanning, shared utilities, LLM routing, and agent orchestration.
Use Cases
- Agent developers can add regression checks when prompts, models, or workflows change.
- RAG teams can test whether generated answers are grounded in retrieved context and meet quality expectations.
- AI safety and security teams can probe an agent for prompt-injection or harmful-output vulnerabilities before deployment.
- Teams with multi-step assistants can evaluate complete conversations and trace-dependent behavior.
Getting Started
Requires Python 3.12 or newer. Install the base package, or add the scan extra for vulnerability and quality scanning:
pip install giskard
pip install "giskard[scan]"
See the repository README and documentation for configuration and examples.
Alternatives
- AgentEvals: AgentEvals focuses on evaluating agent trajectories with ready-made evaluators, while Giskard also targets broader AI security and vulnerability testing.
- Agent Anvil: Agent Anvil centers on CI scenario suites and failure analysis for tool-using agents, while Giskard also probes AI systems for security vulnerabilities.
- Promptfoo: Promptfoo uses declarative configs for evaluating and red-teaming prompts, agents, and RAG apps, while Giskard provides a Python toolkit for repeatable checks.
- EasyJailbreak: EasyJailbreak focuses on assembling and benchmarking jailbreak attacks, while Giskard covers broader agent evaluation and AI vulnerability testing.
| Project | Language | License | Stars | Status |
|---|---|---|---|---|
| giskard-oss | Python | Apache-2.0 | 5.9k | Not checked yet |
| agentevals | Python | MIT | 737 | Active |
| agent-anvil | Python | MIT | 9 | New |
| promptfoo | TypeScript | MIT | 26.4k | Active |
| EasyJailbreak | Python | GPL-3.0 | 914 | Maintained |
Considerations
- LLM-as-judge checks and scan generators require a configured provider SDK and API key.
- The current v3 release is a rewrite focused on dynamic, multi-turn agent testing and has a Python 3.12+ requirement.
- Giskard v2 remains available but is no longer actively maintained. Automatic tabular/ML scanning is v2-only and is not planned for v3.
- Optional aggregated telemetry is described in the project documentation. The README provides environment variables to opt out.
Found this useful?
Share it with someone who would like giskard-oss.