LLMSanitize: An Open-Source Library for Contamination Detection in NLP and LLM Datasets

Introduction

LLMSanitize is an essential open-source Python library dedicated to identifying and mitigating data contamination in Natural Language Processing (NLP) datasets and Large Language Models (LLMs). Data contamination can significantly impact the reliability and trustworthiness of LLM evaluations, making tools like LLMSanitize vital for ensuring robust research and development. The library provides a unified framework to apply various state-of-the-art detection methods.

Installation

LLMSanitize is designed and tested with Python 3.9 and CUDA 11.8. Follow these steps to get started:

First, ensure you have CUDA 11.8 installed. Then, create a conda environment with Python 3.9:

conda create --name llmsanitize python=3.9

Activate your new environment:

conda activate llmsanitize

Finally, install LLMSanitize from PyPI:

pip install llmsanitize

Note that the library specifically uses vllm 0.3.3.

Examples

LLMSanitize supports a wide array of contamination detection methods. For methods requiring vLLM, you first need to launch a vLLM instance.

To launch the vLLM instance, run the following command in a terminal, specifying a port number and model name in the vllm_hosting.sh script:

sh llmsanitize/scripts/vllm_hosting.sh

You can run contamination detection using the provided test scripts. For instance, to run sharded-likelihood on Hellaswag with Llama-2-7B:

sh llmsanitizescripts/tests/closed_data/sharded-likelihood/test_hellaswag.sh -m <path_to_your_llama-2-7b_folder>

For methods using vLLM, such as guided-prompting, pass the port number as an argument:

sh llmsanitizescripts/tests/closed_data/guided-prompting/test_hellaswag.sh -m <path_to_your_llama-2-7b_folder> -p <port_number_from_your_vllm_instance>

Alternatively, you can integrate LLMSanitize directly into your Python scripts:

from llmsanitize import ClosedDataContaminationChecker
args = # setup your argparse here
contamination_checker = ClosedDataContaminationChecker(args)
contamination_checker.run_contamination("guided-prompting") # make sure that your args contain all parameters relevant this specific method

Why Use LLMSanitize?

LLMSanitize offers a robust solution for a critical problem in LLM development. By providing a comprehensive suite of contamination detection methods, it empowers researchers and practitioners to:

• Ensure Data Integrity: Verify the cleanliness of datasets used for training and evaluation.
• Improve Model Reliability: Build more trustworthy LLMs by identifying and addressing contaminated data.
• Support Diverse Methods: Access various detection techniques, including string matching, embedding similarity, model likelihood, and LLM-based approaches, catering to different use cases and model access types.
• Facilitate Research: Serve as a valuable tool for academic and industrial research into LLM data quality and evaluation.

Links

• GitHub Repository: LLMSanitize
• Citation Paper: How Much are LLMs Contaminated? A Comprehensive Survey and the LLMSanitize Library