apache/gluten

Health v2 · maintenance only

Adoption not part of the score

1603 stars · 662 forks observed · 2026-10-12

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Apache Gluten is a middle-layer plugin that offloads JVM-based SQL engine execution (primarily Spark SQL) to high-performance native engines like Velox and ClickHouse. It translates Spark physical plans into Substrait plans executed natively via JNI, delivering up to 3x+ speedups with zero changes to user queries or DataFrame code.

Use cases

• accelerate spark sql queries without changing application code
• speed up etl pipelines running on spark
• offload jvm sql execution to native vectorized engines
• run tpc-h and tpc-ds benchmarks faster on spark
• switch between velox and clickhouse backends for spark workloads
• add columnar shuffle and columnar processing to spark
• reduce compute cost of petabyte-scale spark jobs

When to choose

• you run Spark SQL workloads and want major performance gains without rewriting queries
• your bottleneck is per-operator execution performance rather than query planning
• you want to leverage native engines like Velox or ClickHouse while keeping the Spark ecosystem
• you need automatic fallback to vanilla Spark for unsupported operators

When to avoid

• you don't use a JVM-based SQL engine like Spark
• your queries rely heavily on operators unsupported by the native backends, causing frequent fallback
• you cannot enable off-heap memory or add Spark plugins in your environment
• you need a lightweight solution and can't tolerate the added operational complexity of native bundle JARs

Facets

plugin · maturity active

database streaming developer-tools big-data databases analytics performance jvm cloud spark spark-sql velox clickhouse substrait vectorized-execution columnar jni native-execution query-acceleration simd apache-arrow scala data-engineering linux docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="apache/gluten")