{"name":"Judgy: Correcting LLM Judge Bias for Reliable AI Model Evaluation","description":"Judgy is a Python package designed to improve the reliability of evaluations performed by LLM-as-Judges. It provides tools to estimate the true success rate of a system by correcting for LLM judge bias and generating confidence intervals through bootstrapping. This helps ensure more accurate and trustworthy assessments of AI model performance.","github":"https://github.com/ai-evals-course/judgy","url":"https://osrepos.com/repo/ai-evals-course-judgy","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/ai-evals-course-judgy","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/ai-evals-course-judgy.md","json":"https://osrepos.com/repo/ai-evals-course-judgy.json","topics":["Python","LLM","AI Evaluation","Bias Correction","Confidence Intervals","Machine Learning","NLP"],"keywords":["Python","LLM","AI Evaluation","Bias Correction","Confidence Intervals","Machine Learning","NLP"],"stars":null,"summary":"Judgy is a Python package designed to improve the reliability of evaluations performed by LLM-as-Judges. It provides tools to estimate the true success rate of a system by correcting for LLM judge bias and generating confidence intervals through bootstrapping. This helps ensure more accurate and trustworthy assessments of AI model performance.","content":"## Introduction\n`judgy` is a Python library developed to address the challenges of using Large Language Models (LLMs) as judges for evaluating other AI models or systems. When LLMs are used in this capacity, their inherent biases and potential errors can significantly impact the reliability and accuracy of the evaluation results. This package provides a robust solution to estimate the true success rate of your system by correcting for these LLM judge biases.\n\n## Installation\nGetting started with `judgy` is straightforward. You can install the package using pip:\nbash\npip install judgy\n\n\n## Examples\nHere's a quick example demonstrating how to use `judgy` to estimate a system's true success rate with confidence intervals:\npython\nimport numpy as np\nfrom judgy import estimate_success_rate\n\n# Your data: 1 = Pass, 0 = Fail\ntest_labels = [1, 1, 0, 0, 1, 0, 1, 0]      # Human labels on test set\ntest_preds = [1, 0, 0, 1, 1, 0, 1, 0]       # LLM judge predictions on test set  \nunlabeled_preds = [1, 1, 0, 1, 0, 1, 0, 1]  # LLM judge predictions on unlabeled data\n\n# Estimate true pass rate with 95% confidence interval\ntheta_hat, lower_bound, upper_bound = estimate_success_rate(\n    test_labels=test_labels,\n    test_preds=test_preds, \n    unlabeled_preds=unlabeled_preds\n)\n\nprint(f\"Estimated true pass rate: {theta_hat:.3f}\")\nprint(f\"95% Confidence interval: [{lower_bound:.3f}, {upper_bound:.3f}]\")\n\n\n## Why Use It\nThe core problem `judgy` solves is the unreliability introduced by LLM judge biases. By implementing a bias correction method, `judgy` helps you obtain a more accurate estimate of your system's true performance. It works by first estimating the LLM judge's accuracy (True Positive Rate and True Negative Rate) using a labeled test set. Then, it applies a correction formula to the observed pass rate from the judge. Finally, it uses bootstrap resampling to quantify the uncertainty and provide a confidence interval, giving you a statistically sound range for your system's true success rate. This ensures that your evaluations are not only corrected for bias but also come with a measure of their statistical robustness.\n\n## Links\nFor more detailed information, documentation, and to contribute, visit the official `judgy` resources:\n*   [GitHub Repository](https://github.com/ai-evals-course/judgy){:target=\"_blank\"}\n*   [PyPI Package](https://pypi.org/project/judgy/){:target=\"_blank\"}","metrics":{"detailViews":3,"githubClicks":9},"dates":{"published":null,"modified":"2025-12-07T16:00:43.000Z"}}