pachyderm/pachyderm

Health v2 · maintenance only

Adoption not part of the score

6308 stars · 579 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Pachyderm is a data-centric pipeline platform that automates data transformations with built-in data versioning and lineage tracking. It runs language-agnostic, parallelized multi-stage pipelines on Kubernetes, using standard object stores with deduplication for storage.

Use cases

• version large datasets and track lineage across pipeline stages
• build automated data pipelines that trigger on data changes
• run parallelized ETL transformations on Kubernetes
• create reproducible CI/CD workflows for data science and ML
• process multi-stage pipelines across any data type on cloud or on-prem
• deduplicate and store pipeline data in object stores

When to choose

• you need immutable data versioning and lineage for complex multi-stage pipelines
• your team runs data workloads on Kubernetes and wants autoscaling and parallelism
• you want language-agnostic pipeline steps and reproducible data transformations
• you need a CI/CD-style trigger system driven by data changes

When to avoid

• you only need simple scheduled batch jobs without data versioning
• you cannot operate a Kubernetes cluster or object storage backend
• you want a lightweight single-machine ETL tool
• you need a fully open-source license without commercial terms

Facets

application · maturity active

etl data-science workflow-automation container-orchestration version-control streaming big-data data-science machine-learning microservices go cloud self-hosted windows data-versioning data-lineage data-pipelines cicd-for-data object-storage parallel-processing reproducibility data-engineering containers kubernetes docker linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="pachyderm/pachyderm")