RAGChecker: Diagnose Retrieval-Augmented Generation Systems

Overview

RAGChecker is a Python framework for evaluating and diagnosing Retrieval-Augmented Generation (RAG) systems. It measures the complete pipeline and separates retrieval issues from generation issues, helping teams understand more than a single overall score can show.

Evaluation uses claim-level entailment checks against a ground-truth answer and retrieved context. It is most useful when you have RAG outputs to inspect and want evidence to guide improvements to a retriever, generator, or both.

Key Features

• Reports overall precision, recall, and F1 metrics for RAG results.
• Provides retriever diagnostics, including claim recall and context precision.
• Provides generator diagnostics, including faithfulness, hallucination, and context utilization.
• Uses claim-level entailment for fine-grained checking.
• Runs from either a command-line interface or Python code.
• Supports evaluation with configurable extractor and checker models.
• Includes a benchmark and meta-evaluation data, with usage guidance in the repository.
• Integrates with LlamaIndex through its evaluation tooling.

Use Cases

• RAG application developers can determine whether weak answers stem from missed evidence or generation behavior.
• ML engineers can compare pipeline changes using consistent overall and component-level metrics.
• Researchers can evaluate RAG systems at claim level and study how automated results relate to human judgments.
• Teams tuning retrieval can use claim recall and context precision to investigate coverage and irrelevant context.

Getting Started

Install the package and its required spaCy English model:

pip install ragchecker
python -m spacy download en_core_web_sm

Provide query results, generated responses, retrieved context, and a ground-truth answer, then run the CLI or use the Python API. See the README for input formats, model configuration, and full examples.

Alternatives

• Ragas: Ragas evaluates and optimizes LLM applications with a broad set of metrics and test generation, while RAGChecker focuses on diagnosing retrieval versus generation quality.

Considerations

• Evaluation requires a ground-truth answer for each query. Results also depend on the quality of the supplied responses and retrieved context.
• Claim extraction and checking use models, so you need to configure compatible model access. The README demonstrates AWS Bedrock models, but that example is not a universal requirement.
• The project reports diagnostic metrics rather than automatically improving a RAG system. Interpreting results and making system changes remain the user's responsibility.
• The repository describes a benchmark dataset and human-annotated meta-evaluation data. Consult their documentation to understand their availability and intended use.