webdataset/webdataset

Health v2 · maintenance only

Adoption not part of the score

3169 stars · 236 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A Python library providing a high-performance sequential I/O system based on tar-shard files for large-scale deep learning training, with strong PyTorch integration. It reads data from local disk or cloud object stores via pipes, enabling efficient streaming pipelines for images, audio, video, and other training samples.

Use cases

• stream large image datasets from S3 or GCS into PyTorch training
• load sharded training data sequentially for high-throughput I/O
• create tar-based dataset shards for deep learning
• train models on datasets too large to fit in memory
• augment and preprocess samples in a streaming pipeline
• read MsgPack or CBOR concatenated data sources

When to choose

• you train PyTorch models on large datasets stored locally or in cloud object stores
• you need sequential I/O performance instead of random-access file reads
• you want a simple, tool-friendly format (plain tar archives) for training data

When to avoid

• you need random access to individual samples without sharding overhead
• your project is not Python/PyTorch based (though other language ports exist)
• you need a full dataset management platform with versioning and metadata

Facets

library · maturity stable

etl streaming file-system machine-learning data-science deep-learning machine-learning files python cross-platform pytorch tar-shards data-loading object-storage data-pipeline data-engineering gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="webdataset/webdataset")