promptfoo vs judges
LLM evaluation tools compared
promptfoo and judges both help teams evaluate LLM behavior. promptfoo focuses on repeatable prompt and model tests plus security scans, while judges provides reusable LLM-based classifiers and graders for scoring outputs.
promptfoo: Evaluate and Red-Team LLM Apps
Promptfoo is a TypeScript CLI and library for evaluating and red-teaming LLM-based apps. Compare prompts and models, scan for vulnerabilities, and run checks in CI/CD, with evaluations performed locally.
judges: Evaluate LLM Outputs with Reusable AI Judges
Databricks judges is a Python library for evaluating language model outputs with reusable LLM-based classifiers and graders. Use its research-backed judges, combine evaluations with a jury, or build a custom judge for your task.
| promptfoo | judges | |
|---|---|---|
| Language | TypeScript | Python |
| License | MIT | Apache-2.0 |
| Stars | 26.4k | 339 |
| Forks | 2.6k | 37 |
| Last analyzed | Oct 24, 2026 | Oct 3, 2026 |
Key differences
- promptfoo covers prompt comparisons, regression testing, red-teaming, and code scans; judges focuses on evaluating inputs and outputs with classifiers and graders.
- promptfoo is written in TypeScript and offers a CLI and Node.js library; judges is a Python library with a CLI for single or batch evaluation.
- promptfoo supports CI/CD workflows and says evaluations run locally, though hosted models may need credentials; judges uses provider integrations and requires model access and credentials.
- promptfoo is licensed under MIT, while judges uses Apache-2.0.
- promptfoo is not archived and is described as actively maintained; judges is archived, so ongoing maintenance should not be assumed.
- promptfoo reports 25.9k stars and 2.5k forks; judges reports 339 stars and 37 forks.
Choose promptfoo if you…
- need repeatable prompt and model comparisons in development or CI/CD.
- want to red-team LLM applications or scan code for LLM-related security issues.
- prefer a TypeScript CLI or Node.js library for evaluation workflows.
Choose judges if you…
- want reusable LLM-based classifiers and graders with scores and reasoning.
- need to combine evaluators with a jury or build a custom judge.
- have labeled examples and feedback for creating a task-specific judge with AutoJudge.
Compare the numbers side by side, or add more projects
This comparison is generated with AI from the OSRepos analyses of both projects. Always check each project's repository and documentation before choosing.