huggingface

datatrove: Build Large-Scale Text Data Pipelines

DataTrove is a Python library for processing, filtering and deduplicating text at large scale with reusable pipeline blocks. Its platform-agnostic executors run locally or on Slurm, while fsspec supports local, remote and cloud storage.

View on GitHub Compare
Stars
3.4k
Forks
328
Stars, last 30 days
+33
Last commit
5 days ago
Contributors
67

Our take

Activeregular commits and releases

Health score

60/100

How it is scored
Activity
55
Community
71
Issues
43
Pull requests
70

+33 stars in the last 30 days

DataTrove is a capable fit for large-scale text preparation, with an active release history and a broad contributor base, though its distributed task model requires care when planning shards and retries.

Good fit if

  • You need reusable, composable Python blocks for processing, filtering or deduplicating large text collections.
  • You want the same pipeline to run locally or on Slurm, with optional Ray or Hugging Face Jobs execution.
  • Your team can use Python 3.11 or newer and values a project with regular commits and releases, 67 contributors, and no reported contributor concentration.

Look elsewhere if

  • You need a turnkey data-preparation application rather than a library for assembling and running pipelines.
  • Your workload depends on automatically splitting a single large input file across tasks. DataTrove assigns each file to one task.
  • You need a mature, non-experimental Hugging Face Jobs executor. The README labels that executor experimental.
All health signals
Last commit2026-10-27 (5 days ago)
Commits, last 90 days28
Releases, last 12 months5 (latest v0.10.1, 2026-09-30)
Contributors67 (top contributor: 48% of commits)
Issues closed, last 90 days4 (typically closed in 29 days)
Pull requests merged, last 90 days29 (typically merged in 0 days)
Project age3 years

Checked on 2026-11-02 with the GitHub API.

Project history

Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.

Selected snapshot

Stars

3,425

Forks

326

Change since Nov 17, 2026: +2 stars·0 forks

Overview

DataTrove is a Python library for processing, filtering and deduplicating large text datasets. It replaces one-off processing scripts with reusable pipeline blocks, and uses fsspec to work with local, remote and cloud file systems.

The same pipeline can run locally or on a Slurm cluster, with Ray and Hugging Face Jobs executors also available. Its task-based design supports parallel processing and resuming incomplete tasks, making it relevant to teams preparing large corpora, including LLM training data.

Key Features

  • Compose pipelines from readers, extractors, filters, writers, statistics, tokenization and deduplication blocks.
  • Represent records as Documents with text, an ID and metadata, and add custom processing functions or blocks.
  • Run pipelines locally or on Slurm; Ray and experimental Hugging Face Jobs executors are also supported.
  • Distribute input files across tasks and skip tasks marked complete when resuming a run.
  • Read and write through fsspec-compatible paths, including cloud and Hugging Face storage.
  • Collect distributed dataset statistics, including text, language, sentence and token metrics.
  • Optionally run inference pipelines for synthetic data generation with supported inference servers and endpoints.

Use Cases

  • Data engineers can extract, filter and save text from web crawls such as Common Crawl, locally or on a Slurm cluster.
  • ML teams can deduplicate and profile large training corpora before building datasets.
  • Researchers can create shuffled or sampled dataset subsets and estimate token counts for large Hugging Face datasets.
  • Teams generating synthetic data can run inference as a pipeline stage and use checkpointing to recover from failed tasks.

What you need

Detected in the repository

  • Python >=3.11 (from pyproject.toml)
  • A test suite and automated checks on GitHub Actions

License in plain words

Apache-2.0permissive

  • Commercial use: yes
  • Modify and redistribute: yes
  • You must keep: the license, the NOTICE file and a note of your changes
  • Share your changes: no
  • Includes an explicit patent grant from the contributors:

A summary, not legal advice: the LICENSE file is what applies.

Getting Started

Requires Python 3.11 or newer. Install the project and its default dependencies with:

uv sync

Choose optional extras for capabilities such as I/O formats, processing, S3, Ray or inference. See the README for setup details and examples.

Considerations

  • Task parallelism depends on having suitably sized input files: DataTrove does not automatically split an individual file among tasks. Changing the task count when resuming a failed run can also change file distribution.
  • The library requires Python 3.11 or newer. Some features require optional dependency groups, and inference workloads may also depend on an external inference server or endpoint.
  • The README describes the Hugging Face Jobs executor as experimental, so teams relying on it should account for possible changes.
  • The project is active, with recent commits and releases, and has contributions from 67 people. In the last 90 days, 29 pull requests were merged; issue closures were slower, with a median time of 29 days. The reported contributor concentration is not high.
  • Apache-2.0 licensing permits broad use, subject to the license's terms.

Found this useful?

Share it with someone who would like datatrove.

Comparisons