Ragas: Supercharge Your LLM Application Evaluations

Introduction

Ragas is an open-source framework designed to help you evaluate and optimize your Large Language Model (LLM) applications. It provides objective metrics, intelligent test generation, and data-driven insights, moving beyond subjective assessments. With Ragas, you can streamline your evaluation workflows and build robust, high-performing LLM applications.

Installation

To get started with Ragas, you can install it directly from PyPI:

pip install ragas

Alternatively, you can install it from the source:

pip install git+https://github.com/vibrantlabsai/ragas

Examples

Ragas offers a quickstart command to set up example projects:

# List available templates
ragas quickstart

# Create a RAG evaluation project
ragas quickstart rag_eval

# Specify where you want to create it.
ragas quickstart rag_eval -o ./my-project

You can also evaluate your LLM application using pre-built metrics. Here's an example using DiscreteMetric to assess summary accuracy:

import asyncio
from openai import AsyncOpenAI
from ragas.metrics import DiscreteMetric
from ragas.llms import llm_factory

# Setup your LLM
client = AsyncOpenAI()
llm = llm_factory("gpt-4o", client=client)

# Create a custom aspect evaluator
metric = DiscreteMetric(
    name="summary_accuracy",
    allowed_values=["accurate", "inaccurate"],
    prompt="""Evaluate if the summary is accurate and captures key information.\n\nResponse: {response}\n\nAnswer with only 'accurate' or 'inaccurate'."""
)

# Score your application's output
async def main():
    score = await metric.ascore(
        llm=llm,
        response="The summary of the text is..."
    )
    print(f"Score: {score.value}")  # 'accurate' or 'inaccurate'
    print(f"Reason: {score.reason}")


if __name__ == "__main__":
    asyncio.run(main())

Make sure your OPENAI_API_KEY environment variable is set. For more details, refer to the Quickstart Guide.

Why Use Ragas?

Ragas stands out as a crucial tool for LLM development due to several key features:

• Objective Metrics: It allows you to evaluate LLM applications with precision using both LLM-based and traditional metrics, moving away from subjective assessments.
• Test Data Generation: Ragas can automatically create comprehensive test datasets, covering a wide range of scenarios, even when you don't have a test dataset ready.
• Seamless Integrations: It works flawlessly with popular LLM frameworks like LangChain and major observability tools, fitting easily into your existing workflows.
• Build Feedback Loops: Leverage production data to continually improve your LLM applications, ensuring continuous optimization and better performance.

Links

Explore Ragas further through these official resources:

• Documentation
• Join Discord Community
• Ragas Blog
• Newsletter
• Careers