Observers: A Lightweight Library for AI Observability in Python

Introduction

Observers is a lightweight Python library dedicated to AI observability. It simplifies the process of tracking interactions with generative AI APIs, such as OpenAI, Hugging Face transformers, and other LLM providers. The library is structured around two core concepts: observers, which wrap AI APIs to track interactions, and stores, which synchronize these observations to different storage backends.

Installation

Getting started with Observers is straightforward. You can install the base SDK using pip:

pip install observers

For integrating with other LLM providers via AISuite or Litellm, install with the respective extra:

pip install observers[aisuite] # or observers[litellm]

If you plan to use OpenTelemetry for tracing, install the OpenTelemetry extra:

pip install observers[opentelemetry]

Examples

Observers allows you to easily wrap your existing LLM clients to start logging interactions. Here's a basic example demonstrating how to wrap an OpenAI client and log requests to a Hugging Face dataset using the default DatasetsStore:

from openai import OpenAI
from observers import wrap_openai

openai_client = OpenAI()

client = wrap_openai(openai_client)

response = client.chat.completions.create(
    model="gpt-4o",
    messages=[{"role": "user", "content": "Tell me a joke."}],
)
print(response)

Supported Observers include OpenAI and compatible APIs, Hugging Face transformers, Hugging Face Inference Client, AISuite, and Litellm. This broad support ensures compatibility with a wide range of generative AI models and services.

Supported Stores offer flexible options for data persistence and analysis. These include Hugging Face Datasets, DuckDB, Argilla, and OpenTelemetry. Each store provides different capabilities for viewing, querying, and annotating your AI interaction data.

Why Use Observers

Observers provides a crucial layer for understanding and improving your AI applications. By automatically tracking interactions, it helps in:

• Debugging and Performance Monitoring: Gain insights into how your LLMs are performing, identify errors, and monitor latency or token usage.
• Data Collection for Fine-tuning: Easily collect interaction data that can be used to fine-tune models or improve prompts.
• Compliance and Auditing: Maintain a clear record of AI model inputs and outputs for compliance or auditing purposes.
• Flexibility: Its modular design allows you to switch between different LLM providers and storage solutions without significant code changes.
• Rich Querying Capabilities: Stores like Hugging Face Datasets and DuckDB allow for powerful SQL-based querying of your observation data, making analysis straightforward.

Links

• GitHub Repository: https://github.com/cfahlgren1/observers
• Hugging Face Datasets Viewer: https://huggingface.co/docs/hub/en/datasets-viewer
• Argilla Quickstart: https://docs.argilla.io/latest/getting_started/quickstart/