ragas: Evaluate and Improve LLM Applications

Overview

Ragas helps developers assess LLM application behavior with repeatable, data-driven evaluations rather than relying only on subjective review. It provides metrics for scoring outputs and tools to generate test datasets when a project does not yet have one.

The project is aimed at teams building or operating LLM applications, including RAG systems. It can fit into evaluation workflows alongside LLM frameworks and observability tools, and can use production data to inform ongoing improvements.

Key Features

• Provides LLM-based and traditional metrics for evaluating application outputs.
• Supports custom evaluations, including discrete scores for a defined aspect of a response.
• Generates test datasets to cover a range of evaluation scenarios.
• Offers a command-line quickstart for creating a RAG evaluation project.
• Integrates with popular LLM frameworks and observability tools, according to the project README.
• Supports feedback loops that use production data to guide improvements.

Use Cases

• RAG developers can measure answer quality and iterate on a retrieval or generation pipeline.
• LLM application teams can create evaluation datasets before they have a suitable test set.
• Engineers can define custom checks for requirements such as summary accuracy.
• Teams operating an LLM product can use production data to inform ongoing evaluation and improvement.

Getting Started

Install from PyPI:

pip install ragas

See the README and documentation for setup and evaluation examples.

Alternatives

• giskard-oss: Giskard focuses on agent behavior and vulnerability testing, while Ragas emphasizes LLM application metrics and generated evaluation data, especially for RAG.
• promptfoo: Promptfoo centers on prompt and model comparisons plus security checks in CLI and CI workflows, while Ragas emphasizes evaluation metrics and test-data generation.

Considerations

• Evaluations that use LLM-based metrics require access to a configured LLM. The README example uses an OpenAI client and requires an OPENAI_API_KEY.
• Results depend on the selected metrics and the relevance of the evaluation data to the application being tested.
• The repository reports that it collects minimal, anonymized usage data. The README documents opting out with RAGAS_DO_NOT_TRACK=true.
• The project is not archived. Its Apache-2.0 license permits use under the terms of that license.