# LLMSanitize: An Open-Source Library for Contamination Detection in NLP and LLM Datasets

This repository profile is provided by osrepos.com, an open source repository discovery platform.

Source: osrepos.com
Repository profile: https://osrepos.com/repo/ntunlp-llmsanitize
Generated for open source discovery and AI-assisted research.

LLMSanitize is an open-source Python library designed for detecting contamination in NLP datasets and Large Language Models (LLMs). It offers a comprehensive suite of methods, ranging from string matching to model likelihood and embedding similarity, to ensure data integrity. This tool is crucial for researchers and developers working with LLMs to maintain the reliability of their models and evaluations.

GitHub: https://github.com/ntunlp/LLMSanitize
OSRepos URL: https://osrepos.com/repo/ntunlp-llmsanitize

## Summary

LLMSanitize is an open-source Python library designed for detecting contamination in NLP datasets and Large Language Models (LLMs). It offers a comprehensive suite of methods, ranging from string matching to model likelihood and embedding similarity, to ensure data integrity. This tool is crucial for researchers and developers working with LLMs to maintain the reliability of their models and evaluations.

## Topics

- Python
- NLP
- LLMs
- Data Contamination
- Machine Learning
- Artificial Intelligence
- Dataset Quality

## Repository Information

Last analyzed by OSRepos: Mon Feb 09 2026 12:01:03 GMT+0000 (Western European Standard Time)
Detail views: 3
GitHub clicks: 9

## Safety Notice

OSRepos shares public repositories for knowledge and discovery only. Review source code, dependencies, licenses, and security implications before running or installing anything.

## Content

## Introduction
LLMSanitize is an essential open-source Python library dedicated to identifying and mitigating data contamination in Natural Language Processing (NLP) datasets and Large Language Models (LLMs). Data contamination can significantly impact the reliability and trustworthiness of LLM evaluations, making tools like LLMSanitize vital for ensuring robust research and development. The library provides a unified framework to apply various state-of-the-art detection methods.

## Installation
LLMSanitize is designed and tested with Python 3.9 and CUDA 11.8. Follow these steps to get started:

First, ensure you have CUDA 11.8 installed. Then, create a conda environment with Python 3.9:
bash
conda create --name llmsanitize python=3.9


Activate your new environment:
bash
conda activate llmsanitize


Finally, install LLMSanitize from PyPI:
bash
pip install llmsanitize

Note that the library specifically uses `vllm 0.3.3`.

## Examples
LLMSanitize supports a wide array of contamination detection methods. For methods requiring vLLM, you first need to launch a vLLM instance.

To launch the vLLM instance, run the following command in a terminal, specifying a port number and model name in the `vllm_hosting.sh` script:
bash
sh llmsanitize/scripts/vllm_hosting.sh


You can run contamination detection using the provided test scripts. For instance, to run `sharded-likelihood` on Hellaswag with Llama-2-7B:
bash
sh llmsanitizescripts/tests/closed_data/sharded-likelihood/test_hellaswag.sh -m <path_to_your_llama-2-7b_folder>


For methods using vLLM, such as `guided-prompting`, pass the port number as an argument:
bash
sh llmsanitizescripts/tests/closed_data/guided-prompting/test_hellaswag.sh -m <path_to_your_llama-2-7b_folder> -p <port_number_from_your_vllm_instance>


Alternatively, you can integrate LLMSanitize directly into your Python scripts:
python
from llmsanitize import ClosedDataContaminationChecker
args = # setup your argparse here
contamination_checker = ClosedDataContaminationChecker(args)
contamination_checker.run_contamination("guided-prompting") # make sure that your args contain all parameters relevant this specific method


## Why Use LLMSanitize?
LLMSanitize offers a robust solution for a critical problem in LLM development. By providing a comprehensive suite of contamination detection methods, it empowers researchers and practitioners to:
*   **Ensure Data Integrity**: Verify the cleanliness of datasets used for training and evaluation.
*   **Improve Model Reliability**: Build more trustworthy LLMs by identifying and addressing contaminated data.
*   **Support Diverse Methods**: Access various detection techniques, including string matching, embedding similarity, model likelihood, and LLM-based approaches, catering to different use cases and model access types.
*   **Facilitate Research**: Serve as a valuable tool for academic and industrial research into LLM data quality and evaluation.

## Links
*   **GitHub Repository**: [LLMSanitize](https://github.com/ntunlp/LLMSanitize){:target="_blank"}
*   **Citation Paper**: [How Much are LLMs Contaminated? A Comprehensive Survey and the LLMSanitize Library](https://arxiv.org/abs/2404.00699){:target="_blank"}