rom1504/img2dataset

Health v2 · maintenance only

Adoption not part of the score

4443 stars · 376 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

A Python tool that downloads large sets of image URLs and packages them into machine learning datasets, with resizing and caption support. It can process 100M URLs in 20 hours on a single machine and powers datasets like LAION-5B.

Use cases

• download images from a list of urls into a dataset
• build an image-text dataset for training models
• download and resize millions of images quickly
• create a laion-style dataset from image url list
• package image urls into webdataset shards
• download mscoco or cc3m image datasets

When to choose

• you need to turn a large url list into a usable image dataset at scale
• you want image+caption pairs for training multimodal models
• you need fast, parallel, resumable image downloading with resizing

When to avoid

• you only need a handful of images and a simple downloader suffices
• you need to scrape and discover image urls rather than download from an existing list
• you need dataset labeling or annotation, not downloading

Facets

cli-tool · maturity active

image-processing etl data-generation machine-learning computer-vision big-data python cli windows image-dataset webdataset image-download multimodal url-to-dataset resizer data-engineering linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="rom1504/img2dataset")