Our take
- Activity
- 19
- Community
- 52
- Issues
- 21
- Pull requests
- 20
-2 stars in the last 30 days
Project history
Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.
Stars
2,818Forks
225Change since Nov 14, 2026: 0 stars·0 forks
Overview
PromptBench is a PyTorch-based framework for assessing large language models through a shared set of APIs for models, datasets, prompting methods, and evaluation protocols. It brings standard benchmark evaluation together with tools for examining how prompts affect results and how models respond to adversarial inputs.
It is intended for researchers and developers who need to compare model performance, investigate prompt behavior, or explore dynamic and multimodal evaluation. The project also includes PromptEval for efficient multi-prompt evaluation and DyVal for generating evaluation samples dynamically.
Key Features
- Provides APIs for building model evaluation pipelines and loading supported datasets.
- Covers language and multimodal datasets and models.
- Includes prompt techniques such as chain-of-thought, emotion prompting, and expert prompting.
- Supports adversarial prompt attacks at character, word, sentence, and semantic levels.
- Offers dynamic evaluation through DyVal.
- Includes PromptEval for estimating performance using a smaller sample of evaluation data.
- Supports analysis such as visualization, attack transferability, and word-frequency analysis.
Use Cases
- Researchers comparing language models across established datasets and evaluation protocols.
- Teams studying whether prompting techniques change model performance on a task.
- Robustness researchers testing how model outputs respond to adversarial prompt variations.
- Evaluation designers generating samples dynamically when static test sets may be contaminated.
- Multimodal researchers assessing supported models on image-and-language benchmarks.
Getting Started
Install the package with:
pip install promptbench
For the latest code, installation requirements, and usage examples, see the repository README. The project also provides documentation and a research paper.
Alternatives
- Promptfoo: Promptfoo focuses on testing and red-teaming LLM applications, agents, and RAG systems, while PromptBench emphasizes model evaluation across datasets and adversarial attacks.
- lighteval: Lighteval centers on benchmark tasks, metrics, and model backends; PromptBench puts more emphasis on prompting methods and adversarial robustness.
- EasyJailbreak: EasyJailbreak specializes in assembling and evaluating jailbreak attacks, while PromptBench covers broader model evaluation, prompting methods, and robustness testing.
| Project | Language | License | Stars | Status |
|---|---|---|---|---|
| promptbench | Python | MIT | 2.8k | Not checked yet |
| promptfoo | TypeScript | MIT | 26.4k | Active |
| lighteval | Python | MIT | 2.5k | Active |
| EasyJailbreak | Python | GPL-3.0 | 914 | Maintained |
Considerations
- The repository is marked as archived, so users should check its current maintenance status before depending on it for ongoing work.
- The README notes that the PyPI package may lag behind repository updates.
- The package installation provides basic dependencies; prompt attacks also require TextAttack.
- Supported model integrations may require access to model services or suitable local compute, depending on the selected model.
- The README describes a broad set of components, so confirm that the specific model, dataset, and evaluation method needed for a project are supported.
Found this useful?
Share it with someone who would like promptbench.