{"name":"MarkLLM: An Open-Source Toolkit for LLM Watermarking","description":"MarkLLM is an open-source toolkit designed to simplify the research and application of watermarking technologies for large language models (LLMs). It offers a unified framework for implementing various watermarking algorithms, alongside robust visualization and comprehensive evaluation tools. This toolkit helps researchers and the broader community understand and assess the authenticity and origin of machine-generated text.","github":"https://github.com/THU-BPM/MarkLLM","url":"https://osrepos.com/repo/thu-bpm-markllm","source":"osrepos.com","sourceDescription":"This repository profile is provided by osrepos.com, an open source repository discovery platform.","repositoryProfile":"https://osrepos.com/repo/thu-bpm-markllm","generatedFor":"open source discovery and AI-assisted research","markdown":"https://osrepos.com/repo/thu-bpm-markllm.md","json":"https://osrepos.com/repo/thu-bpm-markllm.json","topics":["large-language-models","llm","safety","toolkit","trustworthy-ai","watermark","python","machine-learning"],"keywords":["large-language-models","llm","safety","toolkit","trustworthy-ai","watermark","python","machine-learning"],"stars":null,"summary":"MarkLLM is an open-source toolkit designed to simplify the research and application of watermarking technologies for large language models (LLMs). It offers a unified framework for implementing various watermarking algorithms, alongside robust visualization and comprehensive evaluation tools. This toolkit helps researchers and the broader community understand and assess the authenticity and origin of machine-generated text.","content":"## Introduction\n\nMarkLLM is a comprehensive, open-source toolkit developed to advance the research and application of watermarking technologies for Large Language Models (LLMs). As the use of LLMs expands, ensuring the authenticity and origin of machine-generated text has become critically important. MarkLLM addresses this need by providing a unified and extensible platform that simplifies access, understanding, and assessment of various watermarking algorithms.\n\nThe toolkit supports a wide array of watermarking methods, offering a streamlined approach to integrate and expand these techniques. It also includes custom visualization tools to demystify how different algorithms operate and a robust evaluation module with 12 tools to assess detectability, robustness, and impact on text quality. MarkLLM was accepted as an EMNLP 2024 Demo, highlighting its significance in the field.\n\n## Installation\n\nTo get started with MarkLLM, follow these steps:\n\n1.  **Python and PyTorch:** Ensure you have Python 3.10 and PyTorch installed.\n2.  **Install dependencies:**\n\n    bash\n    pip install -r requirements.txt\n    \n\n3.  **Cython files (for EXPEdit or ITSEdit):** If you plan to use the EXPEdit or ITSEdit algorithms, you'll need to compile their Cython files:\n\n    bash\n    python watermark/exp_edit/cython_files/setup.py build_ext --inplace\n    \n\n    Then, move the generated `.so` file into `watermark/exp_edit/cython_files/`.\n\n4.  **Hugging Face Models:** Note that model weights for self-trained watermarking algorithms are stored on Hugging Face's [Generative-Watermark-Toolkits](https://huggingface.co/Generative-Watermark-Toolkits). You should download the necessary models according to the config paths and save them to the `model/` directory before running the code.\n\n## Examples\n\nMarkLLM provides intuitive interfaces for invoking watermarking algorithms, visualizing their mechanisms, and applying evaluation pipelines.\n\n### Invoking Watermarking Algorithms\n\npython\nimport torch\nfrom watermark.auto_watermark import AutoWatermark\nfrom utils.transformers_config import TransformersConfig\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\n\n# Device\ndevice = \"cuda\" if torch.cuda.is_available() else \"cpu\"\n\n# Transformers config\ntransformers_config = TransformersConfig(model=AutoModelForCausalLM.from_pretrained('facebook/opt-1.3b').to(device),\n                                         tokenizer=AutoTokenizer.from_pretrained('facebook/opt-1.3b'),\n                                         vocab_size=50272,\n                                         device=device,\n                                         max_new_tokens=200,\n                                         min_length=230,\n                                         do_sample=True,\n                                         no_repeat_ngram_size=4)\n  \n# Load watermark algorithm\nmyWatermark = AutoWatermark.load('KGW', \n                                 algorithm_config='config/KGW.json',\n                                 transformers_config=transformers_config)\n\n# Prompt\nprompt = 'Good Morning.'\n\n# Generate and detect\nwatermarked_text = myWatermark.generate_watermarked_text(prompt)\ndetect_result = myWatermark.detect_watermark(watermarked_text)\nunwatermarked_text = myWatermark.generate_unwatermarked_text(prompt)\ndetect_result = myWatermark.detect_watermark(unwatermarked_text)\n\n\n### Visualizing Mechanisms\n\nMarkLLM offers visualization tools to understand how watermarks are embedded. Here's an example for the KGW family:\n\npython\nimport torch\nfrom visualize.font_settings import FontSettings\nfrom watermark.auto_watermark import AutoWatermark\nfrom utils.transformers_config import TransformersConfig\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom visualize.visualizer import DiscreteVisualizer\nfrom visualize.legend_settings import DiscreteLegendSettings\nfrom visualize.page_layout_settings import PageLayoutSettings\nfrom visualize.color_scheme import ColorSchemeForDiscreteVisualization\n\n# Load watermark algorithm and get data\n# ... (similar setup as above for myWatermark)\nwatermarked_data = myWatermark.get_data_for_visualization(watermarked_text)\nunwatermarked_data = myWatermark.get_data_for_visualization(unwatermarked_text)\n\n# Init visualizer\nvisualizer = DiscreteVisualizer(color_scheme=ColorSchemeForDiscreteVisualization(),\n                                font_settings=FontSettings(), \n                                page_layout_settings=PageLayoutSettings(),\n                                legend_settings=DiscreteLegendSettings())\n# Visualize and Save\nwatermarked_img = visualizer.visualize(data=watermarked_data, \n                                       show_text=True, \n                                       visualize_weight=True, \n                                       display_legend=True)\nwatermarked_img.save(\"KGW_watermarked.png\")\n\n\n### Applying Evaluation Pipelines\n\nThe toolkit includes pipelines for watermark detection and text quality analysis.\n\npython\nimport torch\nfrom evaluation.dataset import C4Dataset\nfrom watermark.auto_watermark import AutoWatermark\nfrom utils.transformers_config import TransformersConfig\nfrom transformers import AutoModelForCausalLM, AutoTokenizer\nfrom evaluation.tools.text_editor import TruncatePromptTextEditor, WordDeletion\nfrom evaluation.tools.success_rate_calculator import DynamicThresholdSuccessRateCalculator\nfrom evaluation.pipelines.detection import WatermarkedTextDetectionPipeline, UnWatermarkedTextDetectionPipeline, DetectionPipelineReturnType\n\n# Load dataset, device, and transformers config\n# ... (similar setup as above)\n\n# Load watermark algorithm\nmy_watermark = AutoWatermark.load('KGW', \n                                  algorithm_config='config/KGW.json',\n                                  transformers_config=transformers_config)\n\n# Init pipelines\npipeline1 = WatermarkedTextDetectionPipeline(\n    dataset=my_dataset, \n    text_editor_list=[TruncatePromptTextEditor(), WordDeletion(ratio=0.3)],\n    show_progress=True, \n    return_type=DetectionPipelineReturnType.SCORES) \n\npipeline2 = UnWatermarkedTextDetectionPipeline(dataset=my_dataset, \n                                               text_editor_list=[],\n                                               show_progress=True,\n                                               return_type=DetectionPipelineReturnType.SCORES)\n\n# Evaluate\ncalculator = DynamicThresholdSuccessRateCalculator(labels=['TPR', 'F1'], rule='best')\nprint(calculator.calculate(pipeline1.evaluate(my_watermark), pipeline2.evaluate(my_watermark)))\n\n\nMore examples and detailed usage instructions can be found in the `test/` and `evaluation/examples/` directories of the repository, as well as in the provided Jupyter notebooks.\n\n## Why Use MarkLLM?\n\nMarkLLM offers several compelling reasons for researchers and developers working with LLM watermarking:\n\n*   **Unified Framework:** It provides a consistent and extensible platform for implementing and comparing various watermarking algorithms, simplifying development and research.\n*   **Comprehensive Evaluation:** With 12 evaluation tools covering detectability, robustness, and text quality, along with customizable automated pipelines, MarkLLM ensures thorough assessment of watermarking technologies.\n*   **Enhanced Understanding:** Visualization tools offer clear insights into the operational mechanisms of different algorithms, making complex concepts more accessible.\n*   **Active Development:** The toolkit is actively maintained and updated with new watermarking methods and features, reflecting the latest advancements in the field.\n*   **Community Contribution:** It encourages community contributions, fostering a collaborative environment for making text watermarking more accessible.\n\n## Links\n\n*   **GitHub Repository:** [https://github.com/THU-BPM/MarkLLM](https://github.com/THU-BPM/MarkLLM){:target=\"_blank\"}\n*   **Homepage:** [https://generative-watermark.github.io/](https://generative-watermark.github.io/){:target=\"_blank\"}\n*   **Paper (arXiv):** [https://arxiv.org/abs/2405.10051](https://arxiv.org/abs/2405.10051){:target=\"_blank\"}\n*   **Hugging Face Models:** [https://huggingface.co/Generative-Watermark-Toolkits](https://huggingface.co/Generative-Watermark-Toolkits){:target=\"_blank\"}\n*   **EMNLP 2024 Demo:** [https://aclanthology.org/2024.emnlp-demo.7/](https://aclanthology.org/2024.emnlp-demo.7/){:target=\"_blank\"}\n*   **Google Colab:** [https://colab.research.google.com/drive/169MS4dY6fKNPZ7-92ETz1bAm_xyNAs0B?usp=sharing](https://colab.research.google.com/drive/169MS4dY6fKNPZ7-92ETz1bAm_xyNAs0B?usp=sharing){:target=\"_blank\"}\n*   **Video Description:** [https://www.youtube.com/watch?v=QN3BhNvw14E&](https://www.youtube.com/watch?v=QN3BhNvw14E&){:target=\"_blank\"}","metrics":{"detailViews":1,"githubClicks":2},"dates":{"published":null,"modified":"2026-06-23T16:12:29.000Z"}}