# google-research/deduplicate-text-datasets

Repository: https://github.com/google-research/deduplicate-text-datasets
Canonical: https://ross.abutalabs.com/products/deduplicate-text-datasets
Language: Rust
License: Apache-2.0
License Family: permissive
Archived: true
Last push: 2024-07-30T21:44:50+00:00

## Health v2 (maintenance only)
Score: 10/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 35, longevity 100
- inputs: {"age_days": 1875, "days_push": 764, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases, archived
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1270, forks 131 (observed 2026-08-28T04:04:11.796386+00:00)

## What it is
A Rust implementation of ExactSubstr deduplication for language model training datasets, with Python scripts for running deduplication and inspecting results, from the ACL 2022 paper 'Deduplicating Training Data Makes Language Models Better'. It also ships deduplicated document clusters for C4, RealNews, LM1B, and Wiki-4B-en.

## Use cases
- remove duplicate text from an LLM training corpus
- deduplicate a scraped web dataset like C4 before training
- find how often a sequence repeats in a large text dataset
- reduce model memorization by cleaning training data
- inspect duplicate clusters in a language modeling dataset
- speed up language model training by shrinking duplicated data

## When to choose
- you need exact substring deduplication at scale for text corpora
- you want the research-validated method from the Google paper
- you have a large machine (many cores, lots of RAM) for web-scale datasets

## When to avoid
- you need fuzzy/semantic near-duplicate detection out of the box
- you want a polished production data pipeline rather than research code
- you only have a small machine and a web-scale dataset

## Facets
- artifact type: library
- maturity: maintenance
- function: nlp, data-science, etl, parser
- domain: machine-learning, large-language-models
- platform: windows, rust, python, cli
- tags: deduplication, suffix-array, text-corpus-cleaning, research-code, llm-training-data, natural-language-processing, data-engineering, linux, macos

## Member repositories
- google-research/deduplicate-text-datasets (main) score 10

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:11.796386+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T05:03:26.316376+00:00, confidence not recorded.
  - readme: https://github.com/google-research/deduplicate-text-datasets (fetched 2026-08-28T04:04:11.796386+00:00, sha 45dc6e284c14)
- Data as of 2026-08-30T08:39:29.467469+00:00.