# NVIDIA-NeMo/Curator

Scalable data pre processing and curation toolkit for LLMs

Repository: https://github.com/NVIDIA-NeMo/Curator
Canonical: https://ross.abutalabs.com/products/nvidia-nemo-curator
Language: Python
License: Apache-2.0
License Family: permissive
Topics: data-curation, llm, data, data-prep, data-preparation, data-processing, data-processing-pipelines, data-quality, datacuration, datarecipes, deduplication, fast-data-processing, fine-tuning, large-language-models, large-scale-data-processing, llmapps, python, llm-data-quality, semantic-deduplication
Last push: 2026-08-26T19:15:40+00:00

## Health v2 (maintenance only)
Score: 86/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 83, longevity 64
- inputs: {"age_days": 902, "days_push": 7, "days_rel": 37, "gap_med": 64, "n_releases_24m": 12}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1736, forks 319 (observed 2026-08-28T04:05:29.778187+00:00)

## What it is
NVIDIA NeMo Curator is a scalable, GPU-accelerated toolkit for preprocessing and curating large text, image, video, and audio datasets for AI model training. It provides repeatable pipelines for loading, filtering, deduplicating, and transforming data that can run on a laptop or across multi-node Ray clusters.

## Use cases
- deduplicate large web-scale text corpora for LLM training
- filter and quality-score training data for fine-tuning
- build speech and audio datasets with speaker diarization
- curate image datasets with aesthetic and NSFW filtering
- run data preparation pipelines on multi-node Ray or Slurm clusters
- generate synthetic training data with an in-pipeline LLM inference server
- detect languages and classify documents at scale

## When to choose
- you need GPU-accelerated processing of very large multi-modal datasets
- you want repeatable, production-grade data curation pipelines for LLM training
- you need semantic or exact deduplication at scale
- you run on HPC clusters (Slurm) or multi-node Ray setups

## When to avoid
- you only need small-scale data cleaning on a CPU laptop
- you want a no-code GUI tool rather than a Python library
- your pipeline is simple enough for pandas or plain scripts

## Facets
- artifact type: library
- maturity: active
- function: etl, data-science, machine-learning, llm-training, nlp, image-processing, audio-processing, video-processing, streaming
- domain: machine-learning, large-language-models, developer-tools
- platform: python, cloud
- tags: data-curation, deduplication, gpu-accelerated, ray, synthetic-data-generation, fine-tuning, data-quality, multi-modal, data-engineering, natural-language-processing, linux, docker, gpu

## Member repositories
- NVIDIA-NeMo/Curator (main) score 86

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:05:29.778187+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:30:39.997942+00:00, confidence not recorded.
  - readme: https://github.com/NVIDIA-NeMo/Curator (fetched 2026-08-28T04:05:29.778187+00:00, sha c65fba70b6ae)
- Data as of 2026-08-30T08:39:29.467469+00:00.