openllmetry: Observe LLM Applications with OpenTelemetry

Overview

OpenLLMetry is a Python project that extends OpenTelemetry with instrumentation for LLM applications. It helps developers inspect model and vector database activity alongside the rest of an application's telemetry, making it easier to investigate behavior and failures across an AI workflow.

Use it when you want LLM-specific traces in an existing OpenTelemetry-compatible observability system. The repository includes a convenience SDK as well as individual instrumentations that can be added to an application already using OpenTelemetry.

Key Features

• Instruments calls to LLM providers including OpenAI, Anthropic, Gemini, and Bedrock.
• Adds instrumentation for vector databases such as Chroma, Pinecone, Qdrant, and Weaviate.
• Supports frameworks and protocols including LangChain, LlamaIndex, OpenAI Agents, and MCP.
• Exports standard OpenTelemetry data to supported destinations, including Datadog, Grafana, Honeycomb, and the OpenTelemetry Collector.
• Offers a Python SDK for initializing instrumentation, or lets existing OpenTelemetry setups use instrumentations directly.
• Builds on OpenTelemetry instrumentation for other application components, such as databases and API calls.

Use Cases

• LLM application developers can trace model calls to understand errors and activity within an AI request.
• Platform and observability teams can send LLM telemetry to their existing OpenTelemetry-compatible stack.
• Teams using vector databases can inspect database activity alongside model-provider calls.
• Developers integrating frameworks such as LangChain or LlamaIndex can add framework instrumentation to their application telemetry.

Project Facts

• Language: Python
• License: Apache-2.0
• Stars: 7.5k
• Forks: 1.1k
• Topics: artifical-intelligence, datascience, generative-ai, good-first-issue, good-first-issues, help-wanted, llm, llmops, metrics, ml, model-monitoring, monitoring, observability, open-source, open-telemetry, opentelemetry, opentelemetry-python, python
• Archived: false

Getting Started

Install the SDK and initialize it in your application:

pip install traceloop-sdk

from traceloop.sdk import Traceloop

Traceloop.init()

See the getting started guide and the README for configuration and integration details.

Alternatives

• Evidently: Evidently focuses on evaluating and monitoring model quality, drift, and LLM outputs, rather than exporting application traces and metrics through OpenTelemetry.
• agenttrail: agenttrail visualizes local AI coding-agent plans, tool calls, and file changes, rather than providing provider- and framework-level OpenTelemetry instrumentation.

Considerations

• You need an OpenTelemetry-compatible destination and configuration to export and inspect telemetry; the project supports multiple destinations, but setup varies by backend.
• Instrumentation depends on the providers, frameworks, and database integrations used by your application. Check the project's instrumentation documentation for relevant setup details.
• The repository reports 733 open issues, so review current issues and releases when evaluating compatibility or investigating integration problems.