tczhangzhi/pytorch-distributed resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1655 stars · 293 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A collection of PyTorch example scripts demonstrating different distributed/multi-GPU training approaches (DataParallel, torch.distributed, torch.multiprocessing, Apex, Horovod, Slurm) on ImageNet. It also includes benchmark results comparing the speed of these methods on Tesla V100 GPUs.

Use cases

• learn how to do multi-gpu training in pytorch
• compare pytorch distributed training frameworks
• quickstart code for torch.distributed
• benchmark apex vs horovod vs dataparallel
• set up distributed training on a slurm cluster
• speed up model training with multiple gpus

When to choose

• you want copy-paste quickstart examples for PyTorch distributed training
• you need to decide which multi-GPU approach to use based on benchmarks
• you are learning distributed training concepts in PyTorch

When to avoid

• you need a production-ready distributed training framework rather than examples
• you need up-to-date examples for the latest PyTorch APIs
• you need single-GPU training or CPU-only training

Facets

learning-resource · maturity maintenance

machine-learning llm-training benchmarking developer-tools deep-learning machine-learning tutorials gpu-computing python cross-platform pytorch distributed-training quickstart horovod apex imagenet multi-gpu example-code gpu linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="tczhangzhi/pytorch-distributed")