togethercomputer/RedPajama-Data resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

4980 stars · 374 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

RedPajama-Data provides code and pipelines for building RedPajama-V2, an open dataset with over 30 trillion tokens of web text for training large language models. It processes CommonCrawl snapshots with quality signal computation and deduplication steps.

Use cases

• build a large pretraining corpus for an LLM
• reproduce the LLaMA training dataset
• deduplicate web crawl text at scale
• compute quality signals for CommonCrawl documents
• filter multilingual web data for model training

When to choose

• you need a massive open web-scale corpus for LLM pretraining
• you want reproducible pipelines for cleaning and deduplicating CommonCrawl data
• you need quality annotations to filter training documents

When to avoid

• you need a small curated dataset rather than web-scale crawl data
• you lack the storage and compute to process terabytes of CommonCrawl data
• you only need inference tooling rather than training data preparation

Facets

dataset · maturity active

etl data-generation nlp llm-training large-language-models machine-learning python cloud common-crawl llm-training-data data-pipeline deduplication quality-signals open-dataset data-engineering natural-language-processing docker linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="togethercomputer/RedPajama-Data")