data-prep-kit

data-prep-kit: Prepare Data for LLM Applications

Data-Prep-Kit is a toolkit for cleaning, transforming, and enriching unstructured data used in LLM training and RAG pipelines. It offers reusable transforms that run with Python or Ray, from local experiments to larger-scale processing.

HTMLApache-2.0PythonLLMData Science
View on GitHub Compare
Stars
966
Forks
255
Stars, last 30 days
+1

Our take

Health score

52/100

How it is scored
Activity
52
Community
81
Issues
32
Pull requests
44

+1 stars in the last 30 days

  • Slower to close issues than 93% of the projects we track

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

969

Forks

257

Change since Nov 17, 2026: 0 stars·0 forks

Overview

Data-Prep-Kit is a toolkit for preparing unstructured data for LLM applications. Its transforms help developers turn raw language, code, and image data into curated datasets for pre-training, fine-tuning, instruction tuning, or retrieval-augmented generation (RAG).

It is suited to teams that need reusable data-processing steps rather than a single-purpose cleaning script. Transforms can run in Python or, where supported, Ray, and can be composed into pipelines. The project also documents Kubernetes deployments and Tekton pipelines for multi-transform workflows.

Key Features

  • Ingestion transforms for sources such as HTML, web pages, documents, and code archives, with Parquet as a common output format.
  • Data-quality transforms for deduplication, filtering, profiling, language identification, and annotation.
  • Language-specific tools for tasks such as document chunking for RAG, tokenization, PII redaction, and readability scoring.
  • Code transforms for programming-language selection, code-quality annotation, and malware annotation.
  • Image transforms for face, people, and NSFW detection.
  • Python-only and Ray execution options across many transforms, with support varying by module.
  • Examples and recipes for composing transforms into end-to-end data preparation workflows.

Use Cases

  • An LLM development team can clean, filter, and annotate a corpus before fine-tuning or instruction tuning.
  • A RAG team can ingest documents, redact sensitive information, and chunk text before indexing it for retrieval.
  • A dataset maintainer can identify duplicates, assess quality, and profile large language or code collections.
  • A code-data team can classify programming languages and annotate code repositories before training or evaluation.
  • A developer can test a single transform in a Colab notebook before integrating it into a larger workflow.

Getting Started

Install the transforms package and its available extras:

pip install uv
uv pip install 'data-prep-toolkit-transforms[all]'

See the repository README and documentation site for setup, examples, and transform-specific guidance.

Alternatives

  • Lance: Lance provides a columnar storage format for ML data, while data-prep-kit focuses on reusable transforms for cleaning and enriching unstructured data.
  • GraphRAG: GraphRAG builds knowledge graphs and retrieves from them for RAG, while data-prep-kit prepares unstructured data through reusable processing transforms.
ProjectLanguageLicenseStarsStatus
data-prep-kitHTMLApache-2.0966Not checked yet
lanceRustApache-2.07.1kActive
graphragPythonMIT36.4kActive

Considerations

  • The transform catalog is broad, but Python-only and Ray support differs by transform. Check the individual transform documentation before choosing a runtime.
  • The README lists Python 3.10 through 3.13 for the PyPI package. Some transforms may have additional dependencies or setup requirements.
  • The project targets unstructured data and currently lists natural language, code, and images as supported modalities. Select transforms and recipes based on the data and pipeline needs of your application.
  • Scaling and Kubernetes or Tekton deployment involve additional setup beyond installing the package; the project documentation provides the relevant guidance.

Found this useful?

Share it with someone who would like data-prep-kit.

Comparisons