Alluxio/alluxio

Health v2 · maintenance only

Adoption not part of the score

7231 stars · 2933 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Alluxio is an open-source distributed caching and data orchestration platform that sits between compute frameworks (Spark, Presto, Trino, PyTorch) and storage systems (S3, GCS, Azure Blob, HDFS). It caches hot data on local NVMe/SSD of compute nodes, exposing it via POSIX, S3-compatible API, or FSSpec so workloads read at local storage speed instead of repeatedly crossing the network to object storage.

Use cases

• speed up Spark and Presto queries against S3 or HDFS
• cache training datasets on GPU nodes to accelerate PyTorch or TensorFlow epochs
• eliminate model-loading cold starts in inference serving by caching model weights
• accelerate checkpoint writes to cloud object storage with write-back caching
• unify access to multiple storage systems behind a single virtual file system
• reduce cloud egress and storage costs by caching data near compute
• support multi-cloud and hybrid data access for analytics clusters

When to choose

• your analytics or ML workloads repeatedly read the same data from slow remote object storage
• you run data-intensive engines like Spark, Presto, or Trino against S3, GCS, or HDFS and want memory/SSD-speed access
• you need a virtual distributed file system layer unifying multiple underlying storage systems
• you want to reduce S3 request costs and egress traffic in large clusters

When to avoid

• your datasets are small and fit entirely in local storage or your database
• you cannot operate an additional distributed Java service cluster
• you need the enterprise features (tens of billions of files, FUSE POSIX for AI) that are only in the paid edition
• your workload is write-heavy with strict consistency requirements against object storage

Facets

service · maturity stable

caching file-system object-storage data-science etl big-data machine-learning microservices cloud-computing cloud jvm distributed-caching data-orchestration virtual-file-system spark presto trino s3 hdfs gpu-acceleration analytics-acceleration data-engineering linux docker kubernetes

Member repositories

For agents

markdown · JSON · MCP: product_card(name="Alluxio/alluxio")