phoenix: Observe and Evaluate AI Applications

Overview

Arize Phoenix is a self-hosted observability and evaluation platform for applications built with LLMs and AI agents. It collects OpenTelemetry-based traces so developers can inspect application behavior and investigate problems across model calls, retrieval, and agent workflows.

Phoenix also supports the development loop before deployment: teams can create datasets, run evaluations and experiments, and compare prompt or model changes. It is useful when application-level tracing and evaluation need to work across different frameworks, providers, and programming languages.

Key Features

• Capture and inspect LLM application traces using OpenTelemetry-based instrumentation.
• Evaluate responses and retrieval performance with LLM-powered evaluations.
• Create versioned datasets for experimentation, evaluation, and fine-tuning.
• Track experiments that compare changes to prompts, models, and retrieval.
• Use the prompt playground to compare models, adjust parameters, and replay traced calls.
• Integrate with a range of agent frameworks and LLM providers through OpenInference instrumentation.
• Connect coding agents and MCP clients to Phoenix data through its CLI and remote MCP endpoint.

Use Cases

• AI engineers debugging an agent or LLM application can inspect traces to understand runtime behavior and troubleshoot errors.
• Teams building retrieval-augmented generation applications can evaluate retrieval and answer quality against datasets.
• Developers iterating on prompts or model choices can run experiments and compare changes before adopting them.
• Organizations that need control over their observability data can run Phoenix on their own machine or deploy it in containers or the cloud.

Getting Started

Install and start Phoenix locally:

pip install arize-phoenix
phoenix serve

See the README and documentation for tracing setup, deployment options, and integration details.

Alternatives

• LangWatch: LangWatch combines production LLM observability with agent simulations and regression testing, while Phoenix focuses on tracing, evaluation, and troubleshooting.
• Weave by Weights & Biases: Weave provides LLM tracing and evaluation integrated with the Weights & Biases ecosystem, while Phoenix is a self-hosted observability platform.
• Langsmith-sdk: LangSmith SDK connects applications to LangSmith for debugging and evaluation, while Phoenix is a self-hosted platform for tracing and troubleshooting.

Considerations

• Phoenix is self-hosted software. Running it requires choosing and operating an environment, such as a local machine or container deployment.
• Tracing integrations use OpenTelemetry and OpenInference instrumentation. Instrumentation setup varies by framework and provider.
• The repository metadata reports the license as NOASSERTION, while the README describes the software as licensed under Elastic License 2.0. Review the repository's license file and legal terms for your intended use.
• The project offers a separate managed product, Arize AX. Phoenix is the self-hosted option.