EasyEdit: An Easy-to-Use Knowledge Editing Framework for LLMs

Introduction

EasyEdit is a powerful and user-friendly open-source framework developed by zjunlp, designed for knowledge editing in Large Language Models (LLMs). Presented at ACL 2024, EasyEdit addresses the critical need to efficiently update, insert, or erase specific knowledge within pre-trained LLMs. This is essential for correcting factual errors, mitigating biases, and adapting models to new information without undergoing extensive retraining. The framework offers a unified approach to various editing scenarios, methods, and evaluation metrics, making it a comprehensive solution for researchers and developers working with LLMs.

Installation

To get started with EasyEdit, ensure you have Python 3.9 or higher installed. Follow these steps for a quick setup:

git clone https://github.com/zjunlp/EasyEdit.git
conda create -n EasyEdit python=3.10
conda activate EasyEdit
pip install -r requirements.txt

Examples

EasyEdit is designed for modularity and flexibility, allowing users to easily integrate and apply editing methods. Here's a simplified example demonstrating how to use BaseEditor for factual knowledge editing:

Step 1: Define the PLM (Pre-trained Language Model) to be edited.

EasyEdit supports models like T5, GPT-J, GPT-NEO, and Llama, retrievable from HuggingFace. You configure the model in a YAML file (e.g., hparams/MEND/gpt2-xl.yaml).

model_name: gpt2-xl
model_class: GPT2LMHeadModel
tokenizer_class: GPT2Tokenizer
tokenizer_name: gpt2-xl
model_parallel: false # true for multi-GPU editing

Step 2: Choose the appropriate Knowledge Editing Method.

For instance, using the MEND method:

## In this case, we use MEND method, so you should import `MENDHyperParams`
from easyeditor import MENDHyperParams
## Loading config from hparams/MEMIT/gpt2-xl.yaml
hparams = MENDHyperParams.from_hparams('./hparams/MEND/gpt2-xl')

Step 3: Provide the edit descriptor and target.

Define the prompts you want to edit and their desired new outputs.

## edit descriptor: prompt that you want to edit
prompts = [
    'What university did Watts Humphrey attend?',
    'Which family does Ramalinaceae belong to',
    'What role does Denny Herzig play in football?'
]
## You can set `ground_truth` to None !!!(or set to original output)
ground_truth = ['Illinois Institute of Technology', 'Lecanorales', 'defender']
## edit target: expected output
target_new = ['University of Michigan', 'Lamiinae', 'winger']

Step 4: Initialize the BaseEditor.

EasyEdit provides a simple and unified way to init Editor, like huggingface: from_hparams.

## Construct Language Model Editor
from easyeditor import BaseEditor
editor = BaseEditor.from_hparams(hparams)

Step 5: (Optional) Provide data for evaluation.

You can include locality_inputs and portability_inputs to assess the broader impact of the edit.

locality_inputs = {
    'neighborhood':{
        'prompt': ['Joseph Fischhof, the', 'Larry Bird is a professional', 'In Forssa, they understand'],
        'ground_truth': ['piano', 'basketball', 'Finnish']
    }
}

Step 6: Perform the Edit and Evaluation.

The edit function will return metrics and the modified model.

metrics, edited_model, _ = editor.edit(
    prompts=prompts,
    ground_truth=ground_truth,
    target_new=target_new,
    locality_inputs=locality_inputs,
    sequential_edit=False # Set to True for continuous editing
)

For more hands-on experience, EasyEdit provides a handy Jupyter Notebook tutorial that demonstrates editing an LLM's knowledge about the US president.

Why Use EasyEdit

EasyEdit stands out as a comprehensive and efficient solution for knowledge editing in LLMs due to several key advantages:

• Unified Framework: It integrates various editing scenarios (factual, multimodal, safety, personality) and techniques (e.g., ROME, MEMIT, SERAC, MEND) into a single, consistent interface.
• Extensive Model Support: The framework supports a wide range of LLMs, from GPT series to Llama, Baichuan, ChatGLM, and more, including Large Multimodal Models (LMMs).
• Robust Evaluation: EasyEdit provides a detailed evaluation suite with metrics like Reliability, Generalization, Locality, and Portability, ensuring thorough assessment of editing success and side effects.
• Continuous Editing: It supports sequential editing, allowing for multiple knowledge updates while maintaining consistency.
• Active Development & Community: The project is actively maintained, with frequent updates, new methods, and datasets being integrated, fostering a vibrant community.
• Detailed Documentation & Tutorials: With comprehensive READMEs, beginner's guides, and Jupyter Notebook tutorials, getting started and exploring advanced features is straightforward.

Links

• GitHub Repository: https://github.com/zjunlp/EasyEdit
• Paper (ACL 2024): https://arxiv.org/abs/2401.01286
• HuggingFace Demo: https://huggingface.co/spaces/zjunlp/EasyEdit
• KnowEdit Benchmark: https://huggingface.co/datasets/zjunlp/KnowEdit