NVIDIA-NeMo/Curator

Health v2 · maintenance only

Adoption not part of the score

1736 stars · 319 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

NVIDIA NeMo Curator is a scalable, GPU-accelerated toolkit for preprocessing and curating large text, image, video, and audio datasets for AI model training. It provides repeatable pipelines for loading, filtering, deduplicating, and transforming data that can run on a laptop or across multi-node Ray clusters.

Use cases

• deduplicate large web-scale text corpora for LLM training
• filter and quality-score training data for fine-tuning
• build speech and audio datasets with speaker diarization
• curate image datasets with aesthetic and NSFW filtering
• run data preparation pipelines on multi-node Ray or Slurm clusters
• generate synthetic training data with an in-pipeline LLM inference server
• detect languages and classify documents at scale

When to choose

• you need GPU-accelerated processing of very large multi-modal datasets
• you want repeatable, production-grade data curation pipelines for LLM training
• you need semantic or exact deduplication at scale
• you run on HPC clusters (Slurm) or multi-node Ray setups

When to avoid

• you only need small-scale data cleaning on a CPU laptop
• you want a no-code GUI tool rather than a Python library
• your pipeline is simple enough for pandas or plain scripts

Facets

library · maturity active

etl data-science machine-learning llm-training nlp image-processing audio-processing video-processing streaming machine-learning large-language-models developer-tools python cloud data-curation deduplication gpu-accelerated ray synthetic-data-generation fine-tuning data-quality multi-modal data-engineering natural-language-processing linux docker gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="NVIDIA-NeMo/Curator")