microsoft

promptbench: Evaluate LLMs and Test Prompt Robustness

PromptBench is a Python library for evaluating language and multimodal models across datasets, prompting methods, and adversarial attacks. It suits researchers and developers comparing model behavior or studying robustness and dynamic evaluation.

View on GitHub Compare
Stars
2.8k
Forks
222
Stars, last 30 days
-2

Our take

Health score

28/100

How it is scored
Activity
19
Community
52
Issues
21
Pull requests
20

-2 stars in the last 30 days

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

2,818

Forks

225

Change since Nov 14, 2026: 0 stars·0 forks

Overview

PromptBench is a PyTorch-based framework for assessing large language models through a shared set of APIs for models, datasets, prompting methods, and evaluation protocols. It brings standard benchmark evaluation together with tools for examining how prompts affect results and how models respond to adversarial inputs.

It is intended for researchers and developers who need to compare model performance, investigate prompt behavior, or explore dynamic and multimodal evaluation. The project also includes PromptEval for efficient multi-prompt evaluation and DyVal for generating evaluation samples dynamically.

Key Features

  • Provides APIs for building model evaluation pipelines and loading supported datasets.
  • Covers language and multimodal datasets and models.
  • Includes prompt techniques such as chain-of-thought, emotion prompting, and expert prompting.
  • Supports adversarial prompt attacks at character, word, sentence, and semantic levels.
  • Offers dynamic evaluation through DyVal.
  • Includes PromptEval for estimating performance using a smaller sample of evaluation data.
  • Supports analysis such as visualization, attack transferability, and word-frequency analysis.

Use Cases

  • Researchers comparing language models across established datasets and evaluation protocols.
  • Teams studying whether prompting techniques change model performance on a task.
  • Robustness researchers testing how model outputs respond to adversarial prompt variations.
  • Evaluation designers generating samples dynamically when static test sets may be contaminated.
  • Multimodal researchers assessing supported models on image-and-language benchmarks.

Getting Started

Install the package with:

pip install promptbench

For the latest code, installation requirements, and usage examples, see the repository README. The project also provides documentation and a research paper.

Alternatives

  • Promptfoo: Promptfoo focuses on testing and red-teaming LLM applications, agents, and RAG systems, while PromptBench emphasizes model evaluation across datasets and adversarial attacks.
  • lighteval: Lighteval centers on benchmark tasks, metrics, and model backends; PromptBench puts more emphasis on prompting methods and adversarial robustness.
  • EasyJailbreak: EasyJailbreak specializes in assembling and evaluating jailbreak attacks, while PromptBench covers broader model evaluation, prompting methods, and robustness testing.
ProjectLanguageLicenseStarsStatus
promptbenchPythonMIT2.8kNot checked yet
promptfooTypeScriptMIT26.4kActive
lightevalPythonMIT2.5kActive
EasyJailbreakPythonGPL-3.0914Maintained

Considerations

  • The repository is marked as archived, so users should check its current maintenance status before depending on it for ongoing work.
  • The README notes that the PyPI package may lag behind repository updates.
  • The package installation provides basic dependencies; prompt attacks also require TextAttack.
  • Supported model integrations may require access to model services or suitable local compute, depending on the selected model.
  • The README describes a broad set of components, so confirm that the specific model, dataset, and evaluation method needed for a project are supported.

Found this useful?

Share it with someone who would like promptbench.

Comparisons