Our take
- Activity
- 52
- Community
- 81
- Issues
- 32
- Pull requests
- 44
+1 stars in the last 30 days
- Slower to close issues than 93% of the projects we track
Project history
Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.
Stars
969Forks
257Change since Nov 17, 2026: 0 stars·0 forks
Overview
Data-Prep-Kit is a toolkit for preparing unstructured data for LLM applications. Its transforms help developers turn raw language, code, and image data into curated datasets for pre-training, fine-tuning, instruction tuning, or retrieval-augmented generation (RAG).
It is suited to teams that need reusable data-processing steps rather than a single-purpose cleaning script. Transforms can run in Python or, where supported, Ray, and can be composed into pipelines. The project also documents Kubernetes deployments and Tekton pipelines for multi-transform workflows.
Key Features
- Ingestion transforms for sources such as HTML, web pages, documents, and code archives, with Parquet as a common output format.
- Data-quality transforms for deduplication, filtering, profiling, language identification, and annotation.
- Language-specific tools for tasks such as document chunking for RAG, tokenization, PII redaction, and readability scoring.
- Code transforms for programming-language selection, code-quality annotation, and malware annotation.
- Image transforms for face, people, and NSFW detection.
- Python-only and Ray execution options across many transforms, with support varying by module.
- Examples and recipes for composing transforms into end-to-end data preparation workflows.
Use Cases
- An LLM development team can clean, filter, and annotate a corpus before fine-tuning or instruction tuning.
- A RAG team can ingest documents, redact sensitive information, and chunk text before indexing it for retrieval.
- A dataset maintainer can identify duplicates, assess quality, and profile large language or code collections.
- A code-data team can classify programming languages and annotate code repositories before training or evaluation.
- A developer can test a single transform in a Colab notebook before integrating it into a larger workflow.
Getting Started
Install the transforms package and its available extras:
pip install uv
uv pip install 'data-prep-toolkit-transforms[all]'
See the repository README and documentation site for setup, examples, and transform-specific guidance.
Alternatives
- Lance: Lance provides a columnar storage format for ML data, while data-prep-kit focuses on reusable transforms for cleaning and enriching unstructured data.
- GraphRAG: GraphRAG builds knowledge graphs and retrieves from them for RAG, while data-prep-kit prepares unstructured data through reusable processing transforms.
| Project | Language | License | Stars | Status |
|---|---|---|---|---|
| data-prep-kit | HTML | Apache-2.0 | 966 | Not checked yet |
| lance | Rust | Apache-2.0 | 7.1k | Active |
| graphrag | Python | MIT | 36.4k | Active |
Considerations
- The transform catalog is broad, but Python-only and Ray support differs by transform. Check the individual transform documentation before choosing a runtime.
- The README lists Python 3.10 through 3.13 for the PyPI package. Some transforms may have additional dependencies or setup requirements.
- The project targets unstructured data and currently lists natural language, code, and images as supported modalities. Select transforms and recipes based on the data and pipeline needs of your application.
- Scaling and Kubernetes or Tekton deployment involve additional setup beyond installing the package; the project documentation provides the relevant guidance.
Found this useful?
Share it with someone who would like data-prep-kit.