Our take
Activeregular commits and releases- Activity
- 55
- Community
- 71
- Issues
- 43
- Pull requests
- 70
+33 stars in the last 30 days
DataTrove is a capable fit for large-scale text preparation, with an active release history and a broad contributor base, though its distributed task model requires care when planning shards and retries.
Good fit if
- You need reusable, composable Python blocks for processing, filtering or deduplicating large text collections.
- You want the same pipeline to run locally or on Slurm, with optional Ray or Hugging Face Jobs execution.
- Your team can use Python 3.11 or newer and values a project with regular commits and releases, 67 contributors, and no reported contributor concentration.
Look elsewhere if
- You need a turnkey data-preparation application rather than a library for assembling and running pipelines.
- Your workload depends on automatically splitting a single large input file across tasks. DataTrove assigns each file to one task.
- You need a mature, non-experimental Hugging Face Jobs executor. The README labels that executor experimental.
All health signals
| Last commit | 2026-10-27 (5 days ago) |
|---|---|
| Commits, last 90 days | 28 |
| Releases, last 12 months | 5 (latest v0.10.1, 2026-09-30) |
| Contributors | 67 (top contributor: 48% of commits) |
| Issues closed, last 90 days | 4 (typically closed in 29 days) |
| Pull requests merged, last 90 days | 29 (typically merged in 0 days) |
| Project age | 3 years |
Checked on 2026-11-02 with the GitHub API.
Project history
Daily GitHub snapshots since Oct 7, 2026. History covers stars and forks; health signals are not tracked historically.
Stars
3,425Forks
326Change since Nov 17, 2026: +2 stars·0 forks
Overview
DataTrove is a Python library for processing, filtering and deduplicating large text datasets. It replaces one-off processing scripts with reusable pipeline blocks, and uses fsspec to work with local, remote and cloud file systems.
The same pipeline can run locally or on a Slurm cluster, with Ray and Hugging Face Jobs executors also available. Its task-based design supports parallel processing and resuming incomplete tasks, making it relevant to teams preparing large corpora, including LLM training data.
Key Features
- Compose pipelines from readers, extractors, filters, writers, statistics, tokenization and deduplication blocks.
- Represent records as Documents with text, an ID and metadata, and add custom processing functions or blocks.
- Run pipelines locally or on Slurm; Ray and experimental Hugging Face Jobs executors are also supported.
- Distribute input files across tasks and skip tasks marked complete when resuming a run.
- Read and write through fsspec-compatible paths, including cloud and Hugging Face storage.
- Collect distributed dataset statistics, including text, language, sentence and token metrics.
- Optionally run inference pipelines for synthetic data generation with supported inference servers and endpoints.
Use Cases
- Data engineers can extract, filter and save text from web crawls such as Common Crawl, locally or on a Slurm cluster.
- ML teams can deduplicate and profile large training corpora before building datasets.
- Researchers can create shuffled or sampled dataset subsets and estimate token counts for large Hugging Face datasets.
- Teams generating synthetic data can run inference as a pipeline stage and use checkpointing to recover from failed tasks.
What you need
Detected in the repository
- Python >=3.11 (from pyproject.toml)
- A test suite and automated checks on GitHub Actions
License in plain words
Apache-2.0permissive
- Commercial use: yes
- Modify and redistribute: yes
- You must keep: the license, the NOTICE file and a note of your changes
- Share your changes: no
- Includes an explicit patent grant from the contributors:
A summary, not legal advice: the LICENSE file is what applies.
Getting Started
Requires Python 3.11 or newer. Install the project and its default dependencies with:
uv sync
Choose optional extras for capabilities such as I/O formats, processing, S3, Ray or inference. See the README for setup details and examples.
Considerations
- Task parallelism depends on having suitably sized input files: DataTrove does not automatically split an individual file among tasks. Changing the task count when resuming a failed run can also change file distribution.
- The library requires Python 3.11 or newer. Some features require optional dependency groups, and inference workloads may also depend on an external inference server or endpoint.
- The README describes the Hugging Face Jobs executor as experimental, so teams relying on it should account for possible changes.
- The project is active, with recent commits and releases, and has contributions from 67 people. In the last 90 days, 29 pull requests were merged; issue closures were slower, with a median time of 29 days. The reported contributor concentration is not high.
- Apache-2.0 licensing permits broad use, subject to the license's terms.
Found this useful?
Share it with someone who would like datatrove.