facebookresearch/cc_net

Health v2 · maintenance only

Flags: no_releases archived

Adoption not part of the score

1045 stars · 154 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

CCNet is a Python pipeline from Facebook AI Research for downloading, deduplicating, and cleaning Common Crawl web data into high-quality monolingual datasets. It uses language models and perplexity filtering to extract per-language corpora for training multilingual language models.

Use cases

• download and clean Common Crawl snapshots
• extract monolingual corpora from web crawl data
• filter web text by language and perplexity
• deduplicate paragraphs across crawl data
• reproduce the CCNet corpus for multilingual model training
• prepare pretraining data for XLM-R style models

When to choose

• you need large-scale multilingual training corpora from Common Crawl
• you want to reproduce the CCNet or XLM-R data pipeline
• you need language identification and LM-based quality filtering on crawl data

When to avoid

• you only need a small curated dataset rather than web-scale crawling
• you need a maintained tool with recent updates and broad platform support
• you cannot run heavy CPU pipelines or external tools like KenLM and SentencePiece

Facets

library · maturity maintenance

etl nlp data-science web-scraping parser machine-learning big-data python cli common-crawl corpus-mining language-model-filtering dataset-preparation deduplication natural-language-processing data-engineering linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="facebookresearch/cc_net")