# Judgy: Correcting LLM Judge Bias for Reliable AI Model Evaluation

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/ai-evals-course-judgy
Generated for open source discovery and AI-assisted research.

Judgy is a Python package designed to improve the reliability of evaluations performed by LLM-as-Judges. It provides tools to estimate the true success rate of a system by correcting for LLM judge bias and generating confidence intervals through bootstrapping. This helps ensure more accurate and trustworthy assessments of AI model performance.

GitHub: https://github.com/ai-evals-course/judgy
OSRepos URL: https://osrepos.com/repo/ai-evals-course-judgy

## Summary

Judgy is a Python package designed to improve the reliability of evaluations performed by LLM-as-Judges. It provides tools to estimate the true success rate of a system by correcting for LLM judge bias and generating confidence intervals through bootstrapping. This helps ensure more accurate and trustworthy assessments of AI model performance.

## Topics

- Python
- LLM
- AI Evaluation
- Bias Correction
- Confidence Intervals
- Machine Learning
- NLP

## Repository Information

Last analyzed by OSRepos: Sun Dec 07 2025 16:00:43 GMT+0000 (Western European Standard Time)
Detail views: 3
GitHub clicks: 9

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction
`judgy` is a Python library developed to address the challenges of using Large Language Models (LLMs) as judges for evaluating other AI models or systems. When LLMs are used in this capacity, their inherent biases and potential errors can significantly impact the reliability and accuracy of the evaluation results. This package provides a robust solution to estimate the true success rate of your system by correcting for these LLM judge biases.

## Installation
Getting started with `judgy` is straightforward. You can install the package using pip:
bash
pip install judgy


## Examples
Here's a quick example demonstrating how to use `judgy` to estimate a system's true success rate with confidence intervals:
python
import numpy as np
from judgy import estimate_success_rate

# Your data: 1 = Pass, 0 = Fail
test_labels = [1, 1, 0, 0, 1, 0, 1, 0]      # Human labels on test set
test_preds = [1, 0, 0, 1, 1, 0, 1, 0]       # LLM judge predictions on test set  
unlabeled_preds = [1, 1, 0, 1, 0, 1, 0, 1]  # LLM judge predictions on unlabeled data

# Estimate true pass rate with 95% confidence interval
theta_hat, lower_bound, upper_bound = estimate_success_rate(
    test_labels=test_labels,
    test_preds=test_preds, 
    unlabeled_preds=unlabeled_preds
)

print(f"Estimated true pass rate: {theta_hat:.3f}")
print(f"95% Confidence interval: [{lower_bound:.3f}, {upper_bound:.3f}]")


## Why Use It
The core problem `judgy` solves is the unreliability introduced by LLM judge biases. By implementing a bias correction method, `judgy` helps you obtain a more accurate estimate of your system's true performance. It works by first estimating the LLM judge's accuracy (True Positive Rate and True Negative Rate) using a labeled test set. Then, it applies a correction formula to the observed pass rate from the judge. Finally, it uses bootstrap resampling to quantify the uncertainty and provide a confidence interval, giving you a statistically sound range for your system's true success rate. This ensures that your evaluations are not only corrected for bias but also come with a measure of their statistical robustness.

## Links
For more detailed information, documentation, and to contribute, visit the official `judgy` resources:
*   [GitHub Repository](https://github.com/ai-evals-course/judgy){:target="_blank"}
*   [PyPI Package](https://pypi.org/project/judgy/){:target="_blank"}