weave: Trace and Evaluate Generative AI Applications

Overview

Weave is a Python toolkit from Weights & Biases for observing and evaluating generative AI applications. It records inputs, outputs, and nested function calls as traces, helping developers investigate behavior that can otherwise be difficult to reproduce or debug.

It also supports structured evaluations and organizes information across experimentation and production. Weave is a fit for developers who want tracing and evaluation integrated into their application workflow, rather than relying only on ad hoc logs.

Key Features

• Trace decorated functions and capture their inputs and outputs.
• Represent related calls as a trace tree for following application execution.
• Track model interactions alongside validation functions and data transformations.
• Build evaluations for comparing language model use cases.
• Organize information across experimentation, evaluation, and production.
• Integrate tracing into Python code with the weave.op decorator.

Use Cases

• An application developer debugging an unexpected language model response by inspecting the calls and inputs that led to it.
• An ML engineer comparing model or prompt changes using repeatable evaluations.
• A team tracking a workflow that combines model API calls with validation and data transformation steps.
• Developers documenting application behavior across experimentation and production through shared traces and evaluation data.

Getting Started

Requires Python 3.10 or higher and a Weights & Biases account. Install the package:

pip install weave

See the README for initialization, tracing examples, and further details.

Alternatives

• phoenix: Phoenix focuses on self-hosted tracing and troubleshooting, while Weave also organizes runs for repeatable experiment comparisons.
• LangWatch: LangWatch emphasizes agent simulations and regression testing alongside production monitoring; Weave centers on tracing and comparing application runs.
• Langsmith-sdk: LangSmith SDKs provide integrations with LangChain and its platform, while Weave offers a broader run-tracing and evaluation workflow.
• evidently: Evidently spans data quality and ML monitoring with reports and metric tracking, while Weave focuses on tracing and evaluating LLM application runs.

Considerations

• Requires Python 3.10 or higher and a Weights & Biases account.
• The README describes the codebase as undergoing cleanup and says work on the Weave engine and Weave boards is paused while the project focuses on tracing and evaluations.
• The README points contributors to weave/trace and weave/trace_server for tracing, and weave/flow for evaluations.