huggingface/datasets

Health v2 · maintenance only

Adoption not part of the score

21870 stars · 3371 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Hugging Face Datasets is a Python library providing one-line access to hundreds of thousands of public datasets on the Hugging Face Hub across text, image, audio, and video modalities. It offers efficient, memory-mapped data loading and preprocessing built on Apache Arrow, with integrations for PyTorch, TensorFlow, JAX, Pandas, NumPy, and Polars.

Use cases

• load a public dataset for model training in one line of code
• preprocess and tokenize a text dataset for fine-tuning an LLM
• stream a huge dataset that doesn't fit on disk or memory
• convert local CSV, JSON, or Parquet files into an ML-ready dataset
• prepare image or audio datasets with transforms and resampling
• share a dataset with the community on the Hugging Face Hub

When to choose

• you need fast, reproducible access to public ML datasets
• you work with large datasets and want memory-mapped, zero-copy loading
• you train models in PyTorch, TensorFlow, or JAX and need dataloader integration
• you want a unified API for text, image, and audio data preprocessing

When to avoid

• you need a general-purpose ETL tool for non-ML data pipelines
• your data lives in a SQL database and you only need simple queries
• you need distributed data processing beyond single-machine Arrow-based workflows

Facets

library · maturity stable

etl data-science machine-learning deep-learning nlp computer-vision audio-processing sdk machine-learning data-science computer-vision artificial-intelligence large-language-models python cross-platform huggingface datasets apache-arrow dataloader data-preprocessing pytorch tensorflow pandas streaming data-engineering natural-language-processing audio

Member repositories

For agents

markdown · JSON · MCP: product_card(name="huggingface/datasets")