agentic_security: Scan LLMs and Agent Workflows for Vulnerabilities

Overview

Agentic Security is a Python vulnerability scanner and AI red-teaming kit for agent workflows and large language models. It sends attack prompts to an LLM through a configurable HTTP request specification, helping developers, researchers, and security teams find unsafe responses and weaknesses before deployment.

The project supports text, image, and audio probes, and can run scans from its UI or as a CI check. It is an independent scanning tool rather than a model: teams connect it to an API they control and choose datasets and scan settings for their target.

Key Features

• Probe for jailbreaks and unsafe behavior with built-in or custom prompt datasets.
• Fuzz LLMs with varied inputs to surface unexpected responses.
• Run multi-step attack simulations.
• Send text, image, and audio inputs to compatible endpoints.
• Configure a target with an HTTP request template and replaceable prompt field.
• Load local CSV datasets containing a prompt column.
• Run scans from the command line, a UI, or a GitHub Actions workflow.
• Set scan budgets and failure thresholds for CI checks.

Use Cases

• Security teams can run recurring scans against an LLM API before or after a model or prompt change.
• AI application developers can add a scan to CI and fail a check when results exceed their configured failure threshold.
• Researchers can compare model behavior across jailbreak and fuzzing datasets.
• Teams building multimodal assistants can probe image and audio endpoints as well as text interfaces.

Getting Started

Install the package:

pip install agentic_security

Run agentic_security --help to see command options. See the README for configuration, datasets, endpoint formats, and CI setup.

Alternatives

• giskard-oss: Giskard offers repeatable evaluations and adversarial checks for LLM agents and RAG systems, while agentic_security focuses on prompt-based scans against HTTP APIs.
• JailbreakEval: JailbreakEval benchmarks methods for detecting jailbreak compliance across datasets, while agentic_security runs broader vulnerability probes against live APIs.

Considerations

• The project is active, with regular commits and releases, but contributions are highly concentrated in one person. The measured median time to close issues is also lengthy, so do not assume quick resolution for support requests.
• The package requires Python >=3.15,<4.0, which may be newer than the runtime supported by many existing environments.
• The README describes custom integration templates as TBD. Verify that your target API's request and response behavior works with the scanner before relying on it.
• The README lists reinforcement-learning-powered attacks and larger, regularly updated datasets as roadmap goals. Treat these as planned work rather than established capabilities.
• Scans send prompts to the endpoint configured by the user. Review the data handling and credentials for that endpoint before testing sensitive systems.