mlfoundations/dclm resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1467 stars · 132 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

DataComp-LM (DCLM) is a benchmark and dataset suite for curating training data for language models, including a leaderboard and evaluation tooling for comparing data filtering pipelines. It provides large-scale web-crawl datasets, filtering baselines, and standardized CORE/EXTENDED evaluation scores for downstream LM performance.

Use cases

• curate pretraining data for language models
• compare data filtering pipelines for LLM training
• evaluate dataset quality with standardized LM benchmarks
• reproduce DCLM CORE and EXTENDED scores
• build a leaderboard submission for training data selection
• filter Common Crawl data for model training

When to choose

• you are researching or building data curation pipelines for LLM pretraining
• you need a standardized benchmark to compare dataset selection methods
• you want access to curated large-scale web-crawl training datasets

When to avoid

• you need a general-purpose data labeling or annotation tool
• you are fine-tuning rather than pretraining and just need a small instruction dataset
• you need inference or serving infrastructure rather than training data tooling

Facets

dataset · maturity active

machine-learning llm-training data-science etl benchmarking large-language-models machine-learning artificial-intelligence python dataset-curation llm-pretraining benchmark leaderboard data-filtering web-crawl data-engineering linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="mlfoundations/dclm")