meta-pytorch/data

Health v2 · maintenance only

Adoption not part of the score

1263 stars · 179 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

TorchData is a PyTorch library providing scalable, performant data loading utilities, including StatefulDataLoader, a drop-in replacement for torch.utils.data.DataLoader with mid-epoch checkpointing via state_dict/load_state_dict. It also offers torchdata.nodes, a library of composable iterators for building streaming data pipelines.

Use cases

• checkpoint dataloader state mid-epoch for resumable training
• replace PyTorch DataLoader with a stateful version
• build streaming data loading pipelines with composable iterators
• track RNG states and worker buffers across training restarts
• share data loading utilities across PyTorch domain libraries

When to choose

• you need fault-tolerant, resumable PyTorch training loops with mid-epoch checkpointing
• you want composable, streaming-style dataloading primitives in PyTorch
• you're on the PyTorch ecosystem and want the successor to torch.utils.data

When to avoid

• you don't use PyTorch or its DataLoader
• you need a general-purpose ETL tool outside ML training
• you require a stable, long-frozen API since the project is still iterating

Facets

library · maturity active

machine-learning etl streaming data-science machine-learning deep-learning python cross-platform pytorch dataloader checkpointing stateful-dataloader data-pipeline torchdata data-engineering gpu

Member repositories

For agents

markdown · JSON · MCP: product_card(name="meta-pytorch/data")