RAGChecker: Diagnose Retrieval-Augmented Generation Systems

Overview

RAGChecker is a Python framework for assessing and diagnosing Retrieval-Augmented Generation (RAG) systems. It uses claim-level entailment to evaluate generated answers against retrieved context and ground-truth answers, helping teams identify whether weaknesses come from retrieval or generation rather than relying on a single overall score.

The project is aimed at developers and researchers who already have RAG outputs to evaluate. It accepts queries, ground-truth answers, generated responses, and retrieved context, then runs configurable extractor and checker models to produce diagnostic metrics. It also integrates with LlamaIndex.

Key Features

• Reports overall precision, recall, and F1 metrics for a RAG pipeline.
• Provides retriever metrics, including claim recall and context precision.
• Provides generator metrics such as faithfulness, hallucination, context utilization, and noise sensitivity.
• Uses claim-level entailment checks for more granular evaluation.
• Supports evaluation through a command-line interface or Python API.
• Includes a benchmark dataset and meta-evaluation resources in the repository.
• Integrates with LlamaIndex for evaluating RAG applications built with that library.

Use Cases

• A RAG application team can compare pipeline results and determine whether retrieval or generation is the main source of poor answers.
• An engineer can inspect context precision and claim recall when tuning a retriever or changing its retrieved chunks.
• Researchers can evaluate answer faithfulness and hallucination across a collection of RAG results.
• LlamaIndex developers can apply RAGChecker metrics to applications built with that framework.

Getting Started

Install the package and download the spaCy English model:

pip install ragchecker
python -m spacy download en_core_web_sm

For input format, model configuration, and CLI or Python usage, see the README and English tutorial.

Alternatives

• opik: Opik combines RAG evaluation with tracing and production monitoring, while RAGChecker focuses on claim-level diagnosis of retriever and generator performance.
• giskard-oss: Giskard emphasizes repeatable agent checks and adversarial safety testing, while RAGChecker measures RAG performance through claim-level entailment.

Considerations

The repository has had no commits in over a year and no releases in the past year; its measured status is inactive. The project has multiple contributors and is not flagged as concentrated around one contributor, but there has been no recent issue or pull-request activity. The README's example uses AWS Bedrock's Meta Llama 3 70B model for extraction and checking, so teams need to choose and configure suitable models and providers for their own setup. The README describes the benchmark as released in December 2024, while its initial feature list calls it upcoming. The project is licensed under Apache-2.0.