# uber/petastorm

Petastorm library enables single machine or distributed training and evaluation of deep learning models from datasets in Apache Parquet format. It supports ML frameworks such as Tensorflow, Pytorch, and PySpark and can be used from pure Python code.

Repository: https://github.com/uber/petastorm
Canonical: https://ross.abutalabs.com/products/petastorm
Language: Python
License: Apache-2.0
License Family: permissive
Topics: tensorflow, pytorch, deep-learning, machine-learning, sysml, pyspark, pyarrow, parquet, parquet-files
Last push: 2026-01-02T23:08:02+00:00

## Health v2 (maintenance only)
Score: 58/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 60, release rhythm 32, longevity 100
- inputs: {"age_days": 3001, "days_push": 243, "days_rel": 243, "gap_med": null, "n_releases_24m": 1}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1891, forks 285 (observed 2026-08-28T04:05:49.621321+00:00)

## What it is
Petastorm is a Python data access library from Uber that enables single-machine or distributed training and evaluation of deep learning models directly from Apache Parquet datasets. It integrates with TensorFlow, PyTorch, and PySpark, and supports multidimensional arrays and extensible codecs on top of Parquet schemas.

## Use cases
- train deep learning models from parquet datasets
- load parquet data into pytorch or tensorflow
- generate ml datasets with pyspark
- distributed training data loading from spark clusters
- read multidimensional arrays stored in parquet
- stream large datasets to gpu training pipelines

## When to choose
- your training data lives in Apache Parquet and you use TensorFlow, PyTorch, or PySpark
- you need distributed data loading for deep learning from Spark-generated datasets
- you want native support for multidimensional arrays and image codecs in Parquet

## When to avoid
- your data is in CSV, JSON, or other non-Parquet formats
- you need a general-purpose ETL tool rather than ML data loading
- you require a project with frequent updates and active development

## Facets
- artifact type: library
- maturity: maintenance
- function: machine-learning, etl, data-science, serialization
- domain: machine-learning, deep-learning, big-data
- platform: python, cross-platform
- tags: parquet, tensorflow, pytorch, pyspark, pyarrow, data-loading, distributed-training, data-engineering

## Member repositories
- uber/petastorm (main) score 58

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:05:49.621321+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:13:00.882647+00:00, confidence not recorded.
  - readme: https://github.com/uber/petastorm (fetched 2026-08-28T04:05:49.621321+00:00, sha 2652d2a600df)
  - registry_pypi: https://pypi.org/pypi/petastorm/json (fetched 2026-08-29T10:52:09.656980+00:00, sha a26b01823757)
- Data as of 2026-08-30T08:39:29.467469+00:00.