RAG-Anything: The All-in-One Multimodal RAG Framework

Introduction

RAG-Anything is an innovative, all-in-one Multimodal Document Processing RAG system developed by HKUDS. It addresses the limitations of traditional text-focused RAG systems by providing seamless processing and querying across various content modalities, including text, images, tables, equations, and multimedia. Built upon the LightRAG framework, RAG-Anything offers a unified solution for handling complex documents, making it invaluable for academic research, technical documentation, and enterprise knowledge management.

Installation

Getting started with RAG-Anything is straightforward. You can install it via PyPI or from the source.

Option 1: Install from PyPI (Recommended)

# Basic installation
pip install raganything

# With optional dependencies for extended format support
pip install 'raganything[all]' # All optional features

Option 2: Install from Source

First, install uv if you haven't already:

curl -LsSf https://astral.sh/uv/install.sh | sh

Then, clone the repository and install dependencies:

git clone https://github.com/HKUDS/RAG-Anything.git
cd RAG-Anything
uv sync

Optional Dependencies

Office Document Processing: For .doc, .docx, .ppt, .pptx, .xls, .xlsx files, LibreOffice must be installed separately. Refer to the official LibreOffice website for downloads.
- macOS: brew install --cask libreoffice
- Ubuntu/Debian: sudo apt-get install libreoffice
Extended Image Formats (.bmp, .tiff, .gif, .webp): Install with pip install 'raganything[image]'.
Text Files (.txt, .md): Install with pip install 'raganything[text]'.

You can verify your MinerU installation, which is used for document parsing, by running:

python -c "from raganything import RAGAnything; rag = RAGAnything(); print('? MinerU installed properly' if rag.check_parser_installation() else '? MinerU installation issue')"

Examples

RAG-Anything provides flexible ways to process and query your documents. Here are a few key examples.

1. End-to-End Document Processing

This example demonstrates how to process a document and then query its content, including multimodal elements.

import asyncio
from raganything import RAGAnything, RAGAnythingConfig
from lightrag.llm.openai import openai_complete_if_cache, openai_embed
from lightrag.utils import EmbeddingFunc

async def main():
    api_key = "your-api-key"
    config = RAGAnythingConfig(
        working_dir="./rag_storage",
        parser="mineru",
        enable_image_processing=True,
        enable_table_processing=True,
        enable_equation_processing=True,
    )

    llm_model_func = lambda prompt, system_prompt=None, history_messages=[], **kwargs: openai_complete_if_cache(
        "gpt-4o-mini", prompt, system_prompt=system_prompt, history_messages=history_messages, api_key=api_key, **kwargs
    )
    vision_model_func = lambda prompt, system_prompt=None, history_messages=[], image_data=None, messages=None, **kwargs: openai_complete_if_cache(
        "gpt-4o", "", system_prompt=None, history_messages=[], messages=messages if messages else ([{"role": "system", "content": system_prompt}] if system_prompt else []) + [{"role": "user", "content": [{"type": "text", "text": prompt}, {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}}]}], api_key=api_key, **kwargs
    ) if image_data else llm_model_func(prompt, system_prompt, history_messages, **kwargs)
    embedding_func = EmbeddingFunc(
        embedding_dim=3072, max_token_size=8192, func=lambda texts: openai_embed(texts, model="text-embedding-3-large", api_key=api_key)
    )

    rag = RAGAnything(
        config=config, llm_model_func=llm_model_func, vision_model_func=vision_model_func, embedding_func=embedding_func
    )

    await rag.process_document_complete(
        file_path="path/to/your/document.pdf", output_dir="./output", parse_method="auto"
    )

    text_result = await rag.aquery("What are the main findings shown in the figures and tables?", mode="hybrid")
    print("Text query result:", text_result)

    multimodal_result = await rag.aquery_with_multimodal(
        "Explain this formula and its relevance to the document content",
        multimodal_content=[{"type": "equation", "latex": "P(d|q) = \\frac{P(q|d) \\cdot P(d)}{P(q)}", "equation_caption": "Document relevance probability"}],
        mode="hybrid"
    )
    print("Multimodal query result:", multimodal_result)

if __name__ == "__main__":
    asyncio.run(main())

2. Direct Content List Insertion

If you have pre-parsed content, you can directly insert it into RAG-Anything without document parsing. This is useful for external parsers or programmatically generated content.

import asyncio
from raganything import RAGAnything, RAGAnythingConfig
from lightrag.llm.openai import openai_complete_if_cache, openai_embed
from lightrag.utils import EmbeddingFunc

async def insert_content_list_example():
    api_key = "your-api-key"
    config = RAGAnythingConfig(
        working_dir="./rag_storage",
        enable_image_processing=True,
        enable_table_processing=True,
        enable_equation_processing=True,
    )

    llm_model_func = lambda prompt, system_prompt=None, history_messages=[], **kwargs: openai_complete_if_cache(
        "gpt-4o-mini", prompt, system_prompt=system_prompt, history_messages=history_messages, api_key=api_key, **kwargs
    )
    vision_model_func = lambda prompt, system_prompt=None, history_messages=[], image_data=None, messages=None, **kwargs: openai_complete_if_cache(
        "gpt-4o", "", system_prompt=None, history_messages=[], messages=messages if messages else ([{"role": "system", "content": system_prompt}] if system_prompt else []) + [{"role": "user", "content": [{"type": "text", "text": prompt}, {"type": "image_url", "image_url": {"url": f"data:image/jpeg;base64,{image_data}"}}]}], api_key=api_key, **kwargs
    ) if image_data else llm_model_func(prompt, system_prompt, history_messages, **kwargs)
    embedding_func = EmbeddingFunc(
        embedding_dim=3072, max_token_size=8192, func=lambda texts: openai_embed(texts, model="text-embedding-3-large", api_key=api_key)
    )

    rag = RAGAnything(
        config=config, llm_model_func=llm_model_func, vision_model_func=vision_model_func, embedding_func=embedding_func
    )

    content_list = [
        {"type": "text", "text": "This is the introduction section of our research paper.", "page_idx": 0},
        {"type": "image", "img_path": "/absolute/path/to/figure1.jpg", "image_caption": ["Figure 1: System Architecture"], "page_idx": 1},
        {"type": "table", "table_body": "| Method | Accuracy | F1-Score |\n|--------|----------|----------|\n| Ours | 95.2% | 0.94 |", "table_caption": ["Table 1: Performance Comparison"], "page_idx": 2},
        {"type": "equation", "latex": "P(d|q) = \\frac{P(q|d) \\cdot P(d)}{P(q)}", "text": "Document relevance probability formula", "page_idx": 3},
    ]

    await rag.insert_content_list(
        content_list=content_list, file_path="research_paper.pdf", display_stats=True
    )

    result = await rag.aquery(
        "What are the key findings and performance metrics mentioned in the research?",
        mode="hybrid"
    )
    print("Query result:", result)

if __name__ == "__main__":
    asyncio.run(insert_content_list_example())

Why Use RAG-Anything?

RAG-Anything stands out as a powerful solution for multimodal RAG due to its comprehensive features:

End-to-End Multimodal Pipeline: It provides a complete workflow from document ingestion and parsing to intelligent multimodal query answering.
Universal Document Support: Seamlessly processes PDFs, Office documents, images, and diverse file formats.
Specialized Content Analysis: Features dedicated processors for images, tables, mathematical equations, and heterogeneous content types.
Multimodal Knowledge Graph: Automatically extracts entities and discovers cross-modal relationships for enhanced understanding.
Hybrid Intelligent Retrieval: Offers advanced search capabilities spanning textual and multimodal content with contextual understanding.
Unified Framework: Eliminates the need for multiple specialized tools, providing a single, cohesive interface for querying documents with mixed content.

RAG-Anything: The All-in-One Multimodal RAG Framework

Summary

Repository Information

Topics

Use at your own risk