huggingface/datatrove

Health v2 · maintenance only

Adoption not part of the score

3308 stars · 292 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

DataTrove is a Python library from Hugging Face for processing, filtering, and deduplicating large-scale text data through customizable pipeline blocks. Pipelines are platform-agnostic, running locally, on Slurm, or on Ray, making it well suited for preparing LLM training data.

Use cases

• filter and deduplicate large text datasets for LLM training
• process common crawl web data at scale
• extract text from warc/arc/wet files
• run data processing pipelines on a slurm cluster
• build custom data processing pipeline blocks
• generate synthetic training data with LLM inference

When to choose

• you need to clean, filter, or deduplicate massive text corpora
• you want platform-agnostic pipelines that run locally or on Slurm/Ray
• you are preparing pretraining data for large language models
• you need to read web archive formats like warc, arc, or wet

When to avoid

• you only need small-scale data wrangling that pandas handles fine
• you need a GUI or no-code ETL tool
• your data is not text-based

Facets

library · maturity active

etl data-science nlp llm-training streaming machine-learning large-language-models python cli data-pipelines text-processing deduplication slurm web-crawl-processing fsspec data-engineering natural-language-processing linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="huggingface/datatrove")