mosaicml/streaming

Health v2 · maintenance only

Adoption not part of the score

1547 stars · 204 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

StreamingDataset is a Python library from MosaicML for fast, accurate streaming of training data from cloud object storage (S3, GCS, Azure, OCI, Databricks, HuggingFace) into PyTorch training jobs. It is designed for multi-node distributed training of large models, with deterministic shuffling, elastic resumption, and correctness guarantees.

Use cases

• stream training data from s3 to pytorch
• train large models on datasets stored in cloud storage
• distributed multi-node data loading for deep learning
• convert datasets to mds format for streaming
• resume training quickly without re-downloading data
• shuffle large datasets deterministically during training
• load huggingface datasets as streaming shards

When to choose

• you train deep learning models on large datasets in cloud object storage
• you need efficient multi-node distributed data loading with determinism
• you want fast, cheap streaming instead of downloading full datasets
• you use PyTorch and want a drop-in StreamingDataLoader

When to avoid

• you only train on small local datasets where a plain DataLoader suffices
• you need a non-PyTorch framework integration
• you require interactive random access rather than sequential streaming

Facets

library · maturity active

streaming etl machine-learning data-science deep-learning machine-learning cloud-computing python cloud cross-platform pytorch streaming-dataset distributed-training cloud-storage mosaicml mds-format data-loader data-engineering docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="mosaicml/streaming")