apache/celeborn

Health v2 · maintenance only

Adoption not part of the score

1061 stars · 457 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Apache Celeborn is an elastic, high-performance intermediate data service for big data compute engines, focused on managing shuffle and spilled data. It decouples compute from storage of intermediate data via a Master/Worker architecture with push-based shuffle write and merged shuffle read.

Use cases

• speed up spark shuffle in large clusters
• replace spark external shuffle service
• manage shuffle data for flink jobs
• reduce shuffle failures and straggler tasks in mapreduce
• disaggregate compute and shuffle storage
• stabilize etl pipelines with heavy shuffle

When to choose

• you run large-scale Spark, Flink, or Hadoop MapReduce workloads with heavy shuffle
• you need elastic, fault-tolerant shuffle management separated from compute nodes
• you want push-based shuffle to improve stability and performance

When to avoid

• your workloads are small and local shuffle works fine
• you cannot operate an additional Master/Worker cluster service
• you use engines other than Spark, Flink, or MapReduce

Facets

service · maturity active

caching message-queue microservices streaming big-data microservices analytics jvm self-hosted shuffle-service apache-spark apache-flink hadoop-mapreduce push-shuffle intermediate-data map-reduce data-engineering linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="apache/celeborn")