datatrove: Build Large-Scale Text Data Pipelines

Overview

DataTrove is a pipeline framework for turning large collections of text files into cleaned, filtered, deduplicated datasets. It is designed to replace one-off processing scripts with reusable blocks that can be composed into a pipeline and run across different execution environments.

The project is aimed at teams working with large text corpora, especially data engineers and machine-learning practitioners preparing training data. Its main value is the combination of data-processing components, distributed task execution, and resumable jobs, rather than a single turnkey dataset-cleaning recipe.

Key Features

• Compose reader, extractor, filter, statistics, tokenization, deduplication, and writer blocks into pipelines.
• Run the same pipeline locally or through Local, Slurm, or Ray executors.
• Split work into tasks and resume by skipping tasks marked as completed after a relaunch.
• Read and write across local, remote, and cloud-backed filesystems through fsspec.
• Process common text-data formats and extract text from raw sources such as webpage HTML.
• Collect dataset statistics across shards and merge results.
• Add custom processing functions or blocks that operate on DataTrove documents.
• Use optional inference components for synthetic data generation with supported inference servers or compatible endpoints.

Use Cases

• Data teams preparing web-scale text for language-model training can combine extraction, quality filters, deduplication, and output steps in one pipeline.
• Researchers processing Common Crawl or other large file collections can distribute files across Slurm tasks and retry incomplete work.
• Dataset maintainers can profile text characteristics, such as length and language scores, before publishing a cleaned dataset.
• Machine-learning teams can generate synthetic text using the optional inference pipeline, then write results to intermediate storage for further processing.

Project Facts

• Language: Python
• License: Apache-2.0
• Stars: 3.4k
• Forks: 310
• Archived: false

Getting Started

Python 3.10 or newer is required. Install the base project and choose optional dependencies for the formats and execution backends you need:

uv sync

For example, add extras with uv sync --extra processing --extra s3. See the README for installation options, examples, and executor configuration.

Considerations

• DataTrove is a framework, not an end-to-end data-cleaning policy. You need to choose and configure the pipeline steps appropriate for your corpus.
• Effective parallelism depends on having enough input files to distribute across tasks. The README notes that a single file is not automatically split into multiple parts.
• Slurm, Ray, S3, inference, and other capabilities require their corresponding optional dependencies and environment setup.
• The Hugging Face Jobs executor is explicitly experimental and may change or be removed without notice.
• The project requires Python 3.10 or newer.