{"name":"Promptfoo: LLM Evaluation and Red Teaming for AI Applications","description":"Promptfoo is an open-source CLI and library designed for evaluating and red-teaming Large Language Model (LLM) applications. It enables developers to test prompts, agents, and RAGs, compare model performance, and secure AI apps through vulnerability scanning. With simple declarative configs and CI/CD integration, Promptfoo helps ship reliable and secure AI solutions.","github":"https://github.com/promptfoo/promptfoo","url":"https://osrepos.com/repo/promptfoo-promptfoo","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/promptfoo-promptfoo","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/promptfoo-promptfoo.md","json":"https://osrepos.com/repo/promptfoo-promptfoo.json","topics":["llm","evaluation","red-teaming","prompt-engineering","testing","ci-cd","TypeScript","AI"],"keywords":["llm","evaluation","red-teaming","prompt-engineering","testing","ci-cd","TypeScript","AI"],"stars":null,"summary":"Promptfoo is an open-source CLI and library designed for evaluating and red-teaming Large Language Model (LLM) applications. It enables developers to test prompts, agents, and RAGs, compare model performance, and secure AI apps through vulnerability scanning. With simple declarative configs and CI/CD integration, Promptfoo helps ship reliable and secure AI solutions.","content":"## Introduction\n\nPromptfoo is a powerful open-source CLI and library for evaluating and red-teaming Large Language Model (LLM) applications. It helps developers move beyond trial-and-error, enabling them to ship secure and reliable AI apps. Used by organizations like OpenAI and Anthropic, Promptfoo provides a robust framework for testing prompts, agents, and RAGs, as well as comparing the performance of various LLMs such as GPT, Claude, Gemini, and Llama.\n\n## Installation\n\nGetting started with Promptfoo is straightforward. You can install it globally via npm, brew, or pip:\n\nsh\nnpm install -g promptfoo\n\n\nAlternatively, you can use `brew install promptfoo` or `pip install promptfoo`. For quick execution without installation, `npx promptfoo@latest` is also available.\n\nMost LLM providers require an API key. Set yours as an environment variable:\n\nsh\nexport OPENAI_API_KEY=sk-abc123\n\n\n## Examples\n\nOnce installed, you can initialize an example project and run your first evaluation:\n\nsh\npromptfoo init --example getting-started\ncd getting-started\npromptfoo eval\npromptfoo view\n\n\nPromptfoo offers a comprehensive suite of features to streamline your LLM development workflow. You can test prompts and models with automated evaluations, secure your LLM apps with red teaming and vulnerability scanning, and compare models side-by-side across various providers. It also supports automating checks in CI/CD and reviewing pull requests for LLM-related security issues. The tool provides visual web viewers for evaluation matrices, command-line output, and detailed security vulnerability reports.\n\n## Why use Promptfoo?\n\nPromptfoo stands out for several key reasons:\n\n*   **Developer-first**: It's fast, with features like live reload and caching, designed for developer efficiency.\n*   **Private**: LLM evaluations run 100% locally, ensuring your prompts never leave your machine.\n*   **Flexible**: It works seamlessly with any LLM API or programming language.\n*   **Battle-tested**: Promptfoo powers LLM applications serving over 10 million users in production.\n*   **Data-driven**: Make informed decisions based on concrete metrics, not just intuition.\n*   **Open source**: It's MIT licensed, backed by an active and supportive community.\n\n## Links\n\nExplore Promptfoo further with these official resources:\n\n*   [Website](https://www.promptfoo.dev)\n*   [Getting Started](https://www.promptfoo.dev/docs/getting-started/)\n*   [Full Documentation](https://www.promptfoo.dev/docs/intro/)\n*   [Red Teaming Guide](https://www.promptfoo.dev/docs/red-team/)\n*   [CLI Usage](https://www.promptfoo.dev/docs/usage/command-line/)\n*   [Node.js Package](https://www.promptfoo.dev/docs/usage/node-package/)\n*   [Supported Models](https://www.promptfoo.dev/docs/providers/)\n*   [Code Scanning Guide](https://www.promptfoo.dev/docs/code-scanning/)\n*   [Discord Community](https://discord.gg/promptfoo)\n*   [GitHub Repository](https://github.com/promptfoo/promptfoo)","metrics":{"detailViews":4,"githubClicks":3},"dates":{"published":null,"modified":"2026-03-24T12:18:12.000Z"}}