# mosaicml/streaming

A Data Streaming Library for Efficient Neural Network Training

Repository: https://github.com/mosaicml/streaming
Canonical: https://ross.abutalabs.com/products/streaming
Homepage: https://streaming.docs.mosaicml.com
Language: Python
License: Apache-2.0
License Family: permissive
Topics: dataset, deep-learning, machine-learning, neural-network, pytorch, streaming
Last push: 2026-06-25T15:02:27+00:00

## Health v2 (maintenance only)
Score: 70/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 89, release rhythm 28, longevity 100
- inputs: {"age_days": 1546, "days_push": 69, "days_rel": 414, "gap_med": 42, "n_releases_24m": 6}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1547, forks 204 (observed 2026-08-28T04:05:01.672836+00:00)

## What it is
StreamingDataset is a Python library from MosaicML for fast, accurate streaming of training data from cloud object storage (S3, GCS, Azure, OCI, Databricks, HuggingFace) into PyTorch training jobs. It is designed for multi-node distributed training of large models, with deterministic shuffling, elastic resumption, and correctness guarantees.

## Use cases
- stream training data from s3 to pytorch
- train large models on datasets stored in cloud storage
- distributed multi-node data loading for deep learning
- convert datasets to mds format for streaming
- resume training quickly without re-downloading data
- shuffle large datasets deterministically during training
- load huggingface datasets as streaming shards

## When to choose
- you train deep learning models on large datasets in cloud object storage
- you need efficient multi-node distributed data loading with determinism
- you want fast, cheap streaming instead of downloading full datasets
- you use PyTorch and want a drop-in StreamingDataLoader

## When to avoid
- you only train on small local datasets where a plain DataLoader suffices
- you need a non-PyTorch framework integration
- you require interactive random access rather than sequential streaming

## Facets
- artifact type: library
- maturity: active
- function: streaming, etl, machine-learning, data-science
- domain: deep-learning, machine-learning, cloud-computing
- platform: python, cloud, cross-platform
- tags: pytorch, streaming-dataset, distributed-training, cloud-storage, mosaicml, mds-format, data-loader, data-engineering, docker

## Member repositories
- mosaicml/streaming (main) score 70

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:05:01.672836+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T04:30:29.831317+00:00, confidence not recorded.
  - readme: https://github.com/mosaicml/streaming (fetched 2026-08-28T04:05:01.672836+00:00, sha 3a451a0e3ae3)
  - homepage: https://streaming.docs.mosaicml.com (fetched 2026-08-29T11:31:35.825562+00:00, sha 46a9561d4b15)
  - site_page: https://docs.mosaicml.com/projects/streaming/en/stable/getting_started/faqs_and_tips.html (fetched 2026-08-29T11:31:35.834740+00:00, sha 66f118758e12)
- Data as of 2026-08-30T08:39:29.467469+00:00.