{"name":"judges: A Python Library for LLM-as-a-Judge Evaluators","description":"The `judges` library from Databricks provides a concise and powerful way to use and create LLM-as-a-Judge evaluators. It offers a curated set of pre-built judges for various use cases, backed by research, and supports both off-the-shelf usage and custom judge creation. This tool helps developers effectively evaluate the performance and quality of their Large Language Models.","github":"https://github.com/quotient-ai/judges","url":"https://osrepos.com/repo/quotient-ai-judges","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/quotient-ai-judges","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/quotient-ai-judges.md","json":"https://osrepos.com/repo/quotient-ai-judges.json","topics":["Python","LLM","AI Evaluation","Machine Learning","NLP","Model Evaluation","Databricks","Open Source"],"keywords":["Python","LLM","AI Evaluation","Machine Learning","NLP","Model Evaluation","Databricks","Open Source"],"stars":null,"summary":"The `judges` library from Databricks provides a concise and powerful way to use and create LLM-as-a-Judge evaluators. It offers a curated set of pre-built judges for various use cases, backed by research, and supports both off-the-shelf usage and custom judge creation. This tool helps developers effectively evaluate the performance and quality of their Large Language Models.","content":"## Introduction\nThe `judges` library, developed by Databricks, is a compact yet powerful Python tool designed for using and creating LLM-as-a-Judge evaluators. Its primary goal is to provide a curated collection of research-backed LLM evaluators in a low-friction format, suitable for a wide range of use cases. Whether you need off-the-shelf evaluation capabilities or inspiration to build your own custom LLM evaluators, `judges` offers a robust solution. The library supports two main types of judges: Classifiers, which return boolean values, and Graders, which provide numerical or Likert scale scores. It also features a `Jury` object for combining multiple judges and `AutoJudge` for creating custom, task-specific evaluators from datasets.\n\n## Installation\nGetting started with `judges` is straightforward. You can install the core library using pip:\n\nbash\npip install judges\n\n\nIf you plan to use the `AutoJudge` feature for creating custom LLM judges, install it with the `auto` extra:\n\nbash\npip install \"judges[auto]\"\n\n\n## Examples\n\n### Using a Classifier Judge\n`judges` provides various pre-built classifiers to evaluate model outputs. Here's an example using `PollMultihopCorrectness` to check if a model's response is factually correct:\n\npython\nfrom openai import OpenAI\nfrom judges.classifiers.correctness import PollMultihopCorrectness\n\nclient = OpenAI()\n\nquestion = \"What is the name of the rabbit in the following story. Respond with 'I don't know' if you don't know.\"\nstory = \"\"\"\nFig was a small, scruffy dog with a big personality. He lived in a quiet little town where everyone knew his name. Fig loved adventures, and every day he would roam the neighborhood, wagging his tail and sniffing out new things to explore.\n\nOne day, Fig discovered a mysterious trail of footprints leading into the woods. Curiosity got the best of him, and he followed them deep into the trees. As he trotted along, he heard rustling in the bushes and suddenly, out popped a rabbit! The rabbit looked at Fig with wide eyes and darted off.\n\nBut instead of chasing it, Fig barked in excitement, as if saying, \"Nice to meet you!\" The rabbit stopped, surprised, and came back. They sat together for a moment, sharing the calm of the woods.\n\nFrom that day on, Fig had a new friend. Every afternoon, the two of them would meet in the same spot, enjoying the quiet companionship of an unlikely friendship. Fig's adventurous heart had found a little peace in the simple joy of being with his new friend.\n\"\"\"\ninput_prompt = f'{story}\\n\\nQuestion:{question}'\nexpected = \"I don't know\"\n\noutput = client.chat.completions.create(\n    model='gpt-4o-mini',\n    messages=[\n        {'role': 'user', 'content': input_prompt},\n    ],\n).choices[0].message.content\n\ncorrectness = PollMultihopCorrectness(model='openai/gpt-4o-mini')\njudgment = correctness.judge(\n    input=input_prompt,\n    output=output,\n    expected=expected,\n)\nprint(judgment.reasoning)\nprint(judgment.score)\n\n\n### Combining Judges with Jury\nFor more diverse and robust evaluations, you can combine multiple judges using the `Jury` object. This allows for averaging or diversifying judgments.\n\npython\nfrom judges import Jury\nfrom judges.classifiers.correctness import PollMultihopCorrectness, RAFTCorrectness\n\n# Assuming 'input_prompt', 'output', and 'expected' are defined as in the previous example\npoll = PollMultihopCorrectness(model='openai/gpt-4o')\nraft = RAFTCorrectness(model='openai/gpt-4o-mini')\n\njury = Jury(judges=[poll, raft], voting_method=\"average\")\n\nverdict = jury.vote(\n    input=input_prompt,\n    output=output,\n    expected=expected,\n)\nprint(verdict.score)\n\n\n### Creating Custom Judges with AutoJudge\n`AutoJudge` simplifies the creation of custom, task-specific LLM judges by leveraging a labeled dataset and a natural language description of the evaluation task.\n\npython\nfrom judges.classifiers.auto import AutoJudge\n\ndataset = [\n    {\n        \"input\": \"Can I ride a dragon in Scotland?\",\n        \"output\": \"Yes, dragons are commonly seen in the highlands and can be ridden with proper training.\",\n        \"label\": 0,\n        \"feedback\": \"Dragons are mythical creatures; the information is fictional.\",\n    },\n    {\n        \"input\": \"Can you recommend a good hotel in Tokyo?\",\n        \"output\": \"Certainly! Hotel Sunroute Plaza Shinjuku is highly rated for its location and amenities. It offers comfortable rooms and excellent service.\",\n        \"label\": 1,\n        \"feedback\": \"Offers a specific and helpful recommendation.\",\n    },\n]\n\ntask = \"Evaluate responses for accuracy, clarity, and helpfulness.\"\n\nautojudge = AutoJudge.from_dataset(\n    dataset=dataset,\n    task=task,\n    model=\"openai/gpt-4-turbo-2024-04-09\",\n)\n\ninput_ = \"What are the top attractions in New York City?\"\noutput = \"Some top attractions in NYC include the Statue of Liberty and Central Park.\"\n\njudgment = autojudge.judge(input=input_, output=output)\nprint(judgment.reasoning)\nprint(judgment.score)\n\n\n### Building a Custom Judge\nFor complete control, you can define your own judge by inheriting from `BaseJudge` and implementing the `.judge()` method. This allows you to specify custom evaluation criteria and prompts.\n\npython\nfrom textwrap import dedent\nfrom judges.base import BaseJudge, Judgment\n\nclass PolitenessJudge(BaseJudge):\n    \"\"\"\n    A judge that evaluates the politeness and respectfulness of model responses.\n    \"\"\"\n    \n    def judge(\n        self,\n        input: str,\n        output: str = None,\n        expected: str = None,\n    ) -> Judgment:\n        system_prompt = \"You are an expert in communication and social etiquette.\"\n        \n        user_prompt = dedent(\n            f\"\"\"\n            Evaluate whether the following response is polite and respectful.\n            \n            Original question: {input}\n            Response to evaluate: {output}\n            \n            Consider factors like:\n            - Use of courteous language\n            - Respectful tone\n            - Appropriate level of formality\n            - Absence of rude or dismissive language\n            \n            Return \"True\" if the response is polite and respectful, \"False\" otherwise.\n            \"\"\"\n        )\n        \n        reasoning, score = self._judge(\n            user_prompt=user_prompt,\n            system_prompt=system_prompt,\n        )\n        \n        return Judgment(reasoning=reasoning, score=score, score_type=\"boolean\")\n\n# Example usage:\npoliteness_judge = PolitenessJudge(model='openai/gpt-4o-mini')\njudgment = politeness_judge.judge(\n    input=\"Can you help me with my homework?\",\n    output=\"Sure! I'd be happy to help you with your homework. What subject are you working on?\"\n)\nprint(judgment.reasoning)\nprint(judgment.score)\n\n\n### CLI Usage\nThe `judges` library also offers a command-line interface for quick evaluations, supporting both single and batch processing.\n\nbash\njudges PollMultihopCorrectness -m gpt-4o-mini -i '[\n    {\n        \"input\": \"What is the capital of France?\",\n        \"output\": \"The capital of France is Madrid.\",\n        \"expected\": \"The capital of France is Paris.\"\n    },\n    {\n        \"input\": \"What is the capital of Germany?\",\n        \"output\": \"The capital of Germany is Paris.\",\n        \"expected\": \"The capital of Germany is Berlin.\"\n    }\n]'\n\n\n## Why Use `judges`?\nThe `judges` library streamlines the complex task of evaluating Large Language Models. By providing a structured, research-backed approach, it helps developers ensure their LLM applications are performing as expected. Its key advantages include:\n*   **Curated Evaluators**: Access to a set of pre-built, research-backed LLM judges for common evaluation tasks like factual correctness, hallucination, and harmfulness.\n*   **Flexibility**: Easily use off-the-shelf judges, combine them into a `Jury` for diversified results, or create highly customized evaluators with `AutoJudge` or by extending the `BaseJudge` class.\n*   **Structured Output**: Judges return `Judgment` objects with clear reasoning and scores, facilitating automated analysis and feedback loops.\n*   **Ease of Use**: Simple installation and a clear API make it accessible for developers of all experience levels.\n*   **CLI Support**: Integrate LLM evaluation into your workflows directly from the command line.\n\n## Links\n*   **GitHub Repository**: <a href=\"https://github.com/databricks/judges\" target=\"_blank\">databricks/judges</a>\n*   **Discord Community**: <a href=\"https://discord.com/invite/YeJzANpntv\" target=\"_blank\">Join the Discord</a>","metrics":{"detailViews":2,"githubClicks":4},"dates":{"published":null,"modified":"2026-04-14T08:49:46.000Z"}}