{"name":"LLMSanitize: An Open-Source Library for Contamination Detection in NLP and LLM Datasets","description":"LLMSanitize is an open-source Python library designed for detecting contamination in NLP datasets and Large Language Models (LLMs). It offers a comprehensive suite of methods, ranging from string matching to model likelihood and embedding similarity, to ensure data integrity. This tool is crucial for researchers and developers working with LLMs to maintain the reliability of their models and evaluations.","github":"https://github.com/ntunlp/LLMSanitize","url":"https://osrepos.com/repo/ntunlp-llmsanitize","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/ntunlp-llmsanitize","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/ntunlp-llmsanitize.md","json":"https://osrepos.com/repo/ntunlp-llmsanitize.json","topics":["Python","NLP","LLMs","Data Contamination","Machine Learning","Artificial Intelligence","Dataset Quality"],"keywords":["Python","NLP","LLMs","Data Contamination","Machine Learning","Artificial Intelligence","Dataset Quality"],"stars":null,"summary":"LLMSanitize is an open-source Python library designed for detecting contamination in NLP datasets and Large Language Models (LLMs). It offers a comprehensive suite of methods, ranging from string matching to model likelihood and embedding similarity, to ensure data integrity. This tool is crucial for researchers and developers working with LLMs to maintain the reliability of their models and evaluations.","content":"## Introduction\nLLMSanitize is an essential open-source Python library dedicated to identifying and mitigating data contamination in Natural Language Processing (NLP) datasets and Large Language Models (LLMs). Data contamination can significantly impact the reliability and trustworthiness of LLM evaluations, making tools like LLMSanitize vital for ensuring robust research and development. The library provides a unified framework to apply various state-of-the-art detection methods.\n\n## Installation\nLLMSanitize is designed and tested with Python 3.9 and CUDA 11.8. Follow these steps to get started:\n\nFirst, ensure you have CUDA 11.8 installed. Then, create a conda environment with Python 3.9:\nbash\nconda create --name llmsanitize python=3.9\n\n\nActivate your new environment:\nbash\nconda activate llmsanitize\n\n\nFinally, install LLMSanitize from PyPI:\nbash\npip install llmsanitize\n\nNote that the library specifically uses `vllm 0.3.3`.\n\n## Examples\nLLMSanitize supports a wide array of contamination detection methods. For methods requiring vLLM, you first need to launch a vLLM instance.\n\nTo launch the vLLM instance, run the following command in a terminal, specifying a port number and model name in the `vllm_hosting.sh` script:\nbash\nsh llmsanitize/scripts/vllm_hosting.sh\n\n\nYou can run contamination detection using the provided test scripts. For instance, to run `sharded-likelihood` on Hellaswag with Llama-2-7B:\nbash\nsh llmsanitizescripts/tests/closed_data/sharded-likelihood/test_hellaswag.sh -m <path_to_your_llama-2-7b_folder>\n\n\nFor methods using vLLM, such as `guided-prompting`, pass the port number as an argument:\nbash\nsh llmsanitizescripts/tests/closed_data/guided-prompting/test_hellaswag.sh -m <path_to_your_llama-2-7b_folder> -p <port_number_from_your_vllm_instance>\n\n\nAlternatively, you can integrate LLMSanitize directly into your Python scripts:\npython\nfrom llmsanitize import ClosedDataContaminationChecker\nargs = # setup your argparse here\ncontamination_checker = ClosedDataContaminationChecker(args)\ncontamination_checker.run_contamination(\"guided-prompting\") # make sure that your args contain all parameters relevant this specific method\n\n\n## Why Use LLMSanitize?\nLLMSanitize offers a robust solution for a critical problem in LLM development. By providing a comprehensive suite of contamination detection methods, it empowers researchers and practitioners to:\n*   **Ensure Data Integrity**: Verify the cleanliness of datasets used for training and evaluation.\n*   **Improve Model Reliability**: Build more trustworthy LLMs by identifying and addressing contaminated data.\n*   **Support Diverse Methods**: Access various detection techniques, including string matching, embedding similarity, model likelihood, and LLM-based approaches, catering to different use cases and model access types.\n*   **Facilitate Research**: Serve as a valuable tool for academic and industrial research into LLM data quality and evaluation.\n\n## Links\n*   **GitHub Repository**: [LLMSanitize](https://github.com/ntunlp/LLMSanitize){:target=\"_blank\"}\n*   **Citation Paper**: [How Much are LLMs Contaminated? A Comprehensive Survey and the LLMSanitize Library](https://arxiv.org/abs/2404.00699){:target=\"_blank\"}","metrics":{"detailViews":3,"githubClicks":9},"dates":{"published":null,"modified":"2026-02-09T12:01:03.000Z"}}