Airweave: Context Retrieval for AI Agents Across Apps and Databases

Introduction

Airweave is a fully open-source context retrieval layer designed to empower AI agents by providing them with access to information across various applications and databases. It connects seamlessly to productivity tools, databases, and document stores, transforming their diverse contents into semantically searchable knowledge bases. This platform offers a standardized interface, accessible via REST API or MCP, handling everything from authentication and data extraction to embedding and serving, making it easier for agents to find relevant information.

Installation

You can get started with Airweave quickly by self-hosting it using Docker and Docker Compose.

First, ensure Docker and Docker Compose are installed on your system.

# 1. Clone the repository
git clone https://github.com/airweave-ai/airweave.git
cd airweave

# 2. Build and run
chmod +x start.sh
./start.sh

Once the setup is complete, you can access the Airweave dashboard at http://localhost:8080. For a managed service option, consider Airweave Cloud.

Examples

Airweave provides SDKs for both Python and TypeScript/JavaScript, allowing developers to easily integrate its powerful search capabilities into their applications.

Python SDK

Install the SDK:

pip install airweave-sdk

Example usage:

from airweave import AirweaveSDK

# Initialize client
client = AirweaveSDK(
    api_key="YOUR_API_KEY",
    base_url="http://localhost:8001"
)

# Create a collection
collection = client.collections.create(name="My Collection")

# Add a source connection (e.g., Stripe)
source = client.source_connections.create(
    name="My Stripe Connection",
    short_name="stripe",
    readable_collection_id=collection.readable_id,
    authentication={
        "credentials": {"api_key": "your_stripe_api_key"}
    }
)

# Semantic search (default)
results = client.collections.search(
    readable_id=collection.readable_id,
    query="Find recent failed payments"
)

# Hybrid search (semantic + keyword)
results = client.collections.search(
    readable_id=collection.readable_id,
    query="customer invoices Q4 2024",
    search_type="hybrid"
)

# Get AI-generated answer instead of raw results
answer = client.collections.search(
    readable_id=collection.readable_id,
    query="What are our customer refund policies?",
    response_type="completion",
    enable_reranking=True
)

TypeScript/JavaScript SDK

Install the SDK:

npm install @airweave/sdk
# or
yarn add @airweave/sdk

Example usage:

import { AirweaveSDKClient, AirweaveSDKEnvironment } from "@airweave/sdk";

// Initialize client
const client = new AirweaveSDKClient({
    apiKey: "YOUR_API_KEY",
    environment: AirweaveSDKEnvironment.Local
});

// Create a collection
const collection = await client.collections.create({
    name: "My Collection"
});

// Add a source connection (e.g., Stripe)
const source = await client.sourceConnections.create({
    name: "My Stripe Connection",
    shortName: "stripe",
    readableCollectionId: collection.readableId,
    authentication: {
        credentials: { apiKey: "your_stripe_api_key" }
    }
});

// Semantic search (default)
const results = await client.collections.search(
    collection.readableId,
    { query: "Find recent failed payments" }
);

// Hybrid search (semantic + keyword)
const hybridResults = await client.collections.search(
    collection.readableId,
    {
        query: "customer invoices Q4 2024",
        searchType: "hybrid"
    }
);

// Get AI-generated answer instead of raw results
const answer = await client.collections.search(
    collection.readableId,
    {
        query: "What are our customer refund policies?",
        responseType: "completion",
        enableReranking: true
    }
);

Why Use Airweave

Airweave stands out as a powerful solution for enhancing AI agent capabilities through robust context retrieval. Its key features include:

Extensive Data Synchronization: Connects to over 30 sources, including popular apps like GitHub, Notion, Slack, Salesforce, and various databases, with minimal configuration.
Intelligent Data Processing: Features entity extraction and a sophisticated transformation pipeline to prepare data for optimal search.
Scalable Architecture: Built with a multi-tenant architecture and OAuth2 support, ensuring security and scalability.
Efficient Updates: Utilizes content hashing for incremental updates, keeping knowledge bases fresh without redundant processing.
Advanced Search Capabilities: Offers semantic search for nuanced agent queries, alongside hybrid search (semantic + keyword), query expansion, reranking, and recency bias options.
Data Versioning: Provides versioning for data changes, ensuring traceability and reliability.

With its comprehensive feature set and support for a wide array of integrations, Airweave simplifies the creation of intelligent AI applications that can effectively leverage vast amounts of distributed information.