uber/petastorm
Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code. observed · 2026-08-28
Health v2 · maintenance only
58/100
- Activity 60
- Release rhythm 32
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 3001
- days_rel: 243
- days_push: 243
- n_releases_24m: 1
Adoption not part of the score
1891 stars · 285 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
Petastorm is a Python data access library from Uber that enables single-machine or distributed training and evaluation of deep learning models directly from Apache Parquet datasets. It integrates with TensorFlow, PyTorch, and PySpark, and supports multidimensional arrays and extensible codecs on top of Parquet schemas.
Use cases
- train deep learning models from parquet datasets
- load parquet data into pytorch or tensorflow
- generate ml datasets with pyspark
- distributed training data loading from spark clusters
- read multidimensional arrays stored in parquet
- stream large datasets to gpu training pipelines
When to choose
- your training data lives in Apache Parquet and you use TensorFlow, PyTorch, or PySpark
- you need distributed data loading for deep learning from Spark-generated datasets
- you want native support for multidimensional arrays and image codecs in Parquet
When to avoid
- your data is in CSV, JSON, or other non-Parquet formats
- you need a general-purpose ETL tool rather than ML data loading
- you require a project with frequent updates and active development
Facets
library · maturity maintenance
machine-learning etl data-science serialization machine-learning deep-learning big-data python cross-platform parquet tensorflow pytorch pyspark pyarrow data-loading distributed-training data-engineering
2 sources
- readme: https://github.com/uber/petastorm · fetched 2026-08-28 · 2652d2a600df
- registry_pypi: https://pypi.org/pypi/petastorm/json · fetched 2026-08-29 · a26b01823757
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| uber/petastorm | main | 58 |
For agents
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem