langwatch: Test and Monitor LLM Applications and AI Agents

Overview

LangWatch is a platform for teams building and operating LLM applications and AI agents. It brings testing and evaluation together with production observability, helping teams inspect model activity and assess agent behavior as systems change.

The platform also covers AI gateway and governance workflows, including provider routing, usage budgets, and visibility into company AI tools. It is aimed at teams that need more than standalone tracing or a one-off evaluation script.

Key Features

• Trace and observe LLM application activity, with integrations for frameworks and model providers.
• Run simulation tests to evaluate or benchmark agents, including voice agents.
• Manage prompts and evaluations as part of LLM operations.
• Track coding-agent sessions and analyze usage costs, including cost per pull request and per team.
• Route model requests through an OpenAI- and Anthropic-compatible AI gateway with virtual keys and budgets.
• Collect information about organizational AI tools and apply governance workflows.
• Integrate with coding assistants, agent frameworks, no-code platforms, and OpenTelemetry.

Use Cases

• AI product teams can test agent behavior before releases and monitor interactions after deployment.
• Engineering teams using coding assistants can inspect sessions and understand usage costs across pull requests or teams.
• Platform teams can give internal users or customers virtual keys with budgets and route requests through a shared gateway.
• IT and governance teams can identify AI tools in use across an organization and review usage or anomalies.

Getting Started

For a self-hosted instance, run:

npx @langwatch/server

Node.js is required. See the README and self-hosting documentation for setup details.

Alternatives

• phoenix: Phoenix focuses on self-hosted tracing, evaluation, and troubleshooting, while LangWatch also includes simulation testing, prompt management, and an AI gateway.
• weave: Weave centers on tracing and evaluating application runs, while LangWatch adds simulation testing, prompt management, an AI gateway, and governance.
• evidently: Evidently is a Python framework for evaluations and monitoring, while LangWatch is a broader platform with observability, simulations, prompt management, and an AI gateway.

Considerations

LangWatch spans several operational areas, so teams should evaluate which parts they need and how the integrations fit their existing stack. Self-hosting requires Node.js, and the README directs production deployments to separate self-hosting guidance. The repository describes its core as Apache 2.0, but notes that enterprise modules under platform/app/ee/ require a commercial license for production use; SDKs are MIT-licensed.