judges: Evaluate LLM Outputs with Reusable AI Judges

Overview

databricks/judges is a Python library for evaluating model inputs and outputs with LLM-as-a-Judge evaluators. It provides a collection of classifiers and graders for tasks such as correctness, hallucination, safety, relevance, and response quality, so teams can apply established evaluation approaches without building every evaluator from scratch.

It is aimed at developers and teams testing or monitoring language model applications. Use an individual judge for a specific criterion, combine judges with a jury, or create a custom judge when the supplied evaluations do not match your task.

Key Features

• Includes classifiers that return boolean judgments and graders that return numerical or Likert-scale scores.
• Returns a judgment with a score and reasoning for each evaluation.
• Covers evaluation categories including factual correctness, hallucination, harmfulness, refusal, relevance, and response quality.
• Combines multiple judges through a Jury and its voting methods.
• Supports custom judges built on BaseJudge.
• Provides AutoJudge to create a task-specific judge from labeled examples and feedback.
• Offers a CLI for single or batch evaluation using JSON input.
• Uses instructor for structured model outputs and supports models through its provider integrations.

Use Cases

• LLM application developers can check generated answers against expected responses during evaluation or testing.
• RAG teams can assess correctness, factuality, or coverage of responses before changing prompts or retrieval settings.
• Safety reviewers can evaluate outputs for harmful content or refusal behavior as part of an assessment workflow.
• Evaluation teams can combine different judges to compare judgments or explore how results vary across evaluators.
• Product teams with labeled examples can use AutoJudge to build a task-specific evaluator from human feedback.

Getting Started

Install the package:

pip install judges

The examples, model setup, and judge API are documented in the README.

Alternatives

• judgy: judgy estimates pass rates from LLM judge predictions and human-labeled calibration data, correcting for judge errors rather than providing reusable graders.

Considerations

• Judgments come from language models, so they are evaluations to validate rather than guaranteed ground truth.
• Model access and credentials are required. The README describes using instructor with provider-specific model setup, including an API key.
• The repository is archived, so ongoing maintenance should not be assumed.
• AutoJudge is an optional extension installed with pip install "judges[auto]" and depends on labeled examples and feedback.