google-research/deduplicate-text-datasets

Health v2 · maintenance only

Flags: no_releases archived

Adoption not part of the score

1270 stars · 131 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A Rust implementation of ExactSubstr deduplication for language model training datasets, with Python scripts for running deduplication and inspecting results, from the ACL 2022 paper 'Deduplicating Training Data Makes Language Models Better'. It also ships deduplicated document clusters for C4, RealNews, LM1B, and Wiki-4B-en.

Use cases

• remove duplicate text from an LLM training corpus
• deduplicate a scraped web dataset like C4 before training
• find how often a sequence repeats in a large text dataset
• reduce model memorization by cleaning training data
• inspect duplicate clusters in a language modeling dataset
• speed up language model training by shrinking duplicated data

When to choose

• you need exact substring deduplication at scale for text corpora
• you want the research-validated method from the Google paper
• you have a large machine (many cores, lots of RAM) for web-scale datasets

When to avoid

• you need fuzzy/semantic near-duplicate detection out of the box
• you want a polished production data pipeline rather than research code
• you only have a small machine and a web-scale dataset

Facets

library · maturity maintenance

nlp data-science etl parser machine-learning large-language-models windows rust python cli deduplication suffix-array text-corpus-cleaning research-code llm-training-data natural-language-processing data-engineering linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="google-research/deduplicate-text-datasets")