Giskard-AI

giskard-oss: Test and Red-Team LLM Agents

Giskard is a Python toolkit for evaluating agent behavior and probing AI systems for vulnerabilities. It suits teams building LLM agents or RAG applications that need repeatable checks, safety testing, and adversarial evaluation.

PythonApache-2.0LLMAI AgentsEvaluation
View on GitHub Compare
Stars
5.9k
Forks
547
Stars, last 30 days
+46

Our take

Health score

77/100

How it is scored
Activity
86
Community
82
Issues
78
Pull requests
61

+46 stars in the last 30 days

  • Releases more often than 88% of the projects we track
  • Merges more pull requests than 88% of the projects we track

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

5,953

Forks

573

Change since Nov 15, 2026: +2 stars·+2 forks

Overview

Giskard is a modular Python toolkit for testing and evaluating LLM-based agents, including multi-turn systems and RAG applications. It addresses the difficulty of validating systems whose outputs can vary, combining scenario-based checks with automated vulnerability scanning.

Use it to define evaluations for a callable system, run checks against its outputs or traces, and generate adversarial test suites. The current v3 packages focus on agent evaluation and security; legacy automatic tabular-model scanning remains in v2.

Key Features

  • Define scenarios and suites for synchronous or asynchronous targets.
  • Run checks such as string matching, regex, semantic similarity, groundedness, and LLM-as-judge assessments.
  • Evaluate multi-turn interactions, not only individual prompts and responses.
  • Scan agents with generated adversarial tests for prompt injection, harmful content, stereotypes, and misinformation.
  • Evaluate RAG answer quality and grounding.
  • Use modular packages for checks, scanning, shared utilities, LLM routing, and agent orchestration.

Use Cases

  • Agent developers can add regression checks when prompts, models, or workflows change.
  • RAG teams can test whether generated answers are grounded in retrieved context and meet quality expectations.
  • AI safety and security teams can probe an agent for prompt-injection or harmful-output vulnerabilities before deployment.
  • Teams with multi-step assistants can evaluate complete conversations and trace-dependent behavior.

Getting Started

Requires Python 3.12 or newer. Install the base package, or add the scan extra for vulnerability and quality scanning:

pip install giskard
pip install "giskard[scan]"

See the repository README and documentation for configuration and examples.

Alternatives

  • AgentEvals: AgentEvals focuses on evaluating agent trajectories with ready-made evaluators, while Giskard also targets broader AI security and vulnerability testing.
  • Agent Anvil: Agent Anvil centers on CI scenario suites and failure analysis for tool-using agents, while Giskard also probes AI systems for security vulnerabilities.
  • Promptfoo: Promptfoo uses declarative configs for evaluating and red-teaming prompts, agents, and RAG apps, while Giskard provides a Python toolkit for repeatable checks.
  • EasyJailbreak: EasyJailbreak focuses on assembling and benchmarking jailbreak attacks, while Giskard covers broader agent evaluation and AI vulnerability testing.
ProjectLanguageLicenseStarsStatus
giskard-ossPythonApache-2.05.9kNot checked yet
agentevalsPythonMIT737Active
agent-anvilPythonMIT9New
promptfooTypeScriptMIT26.4kActive
EasyJailbreakPythonGPL-3.0914Maintained

Considerations

  • LLM-as-judge checks and scan generators require a configured provider SDK and API key.
  • The current v3 release is a rewrite focused on dynamic, multi-turn agent testing and has a Python 3.12+ requirement.
  • Giskard v2 remains available but is no longer actively maintained. Automatic tabular/ML scanning is v2-only and is not planned for v3.
  • Optional aggregated telemetry is described in the project documentation. The README provides environment variables to opt out.

Found this useful?

Share it with someone who would like giskard-oss.

Comparisons