apache/datafusion-comet

Health v2 · maintenance only

Adoption not part of the score

1262 stars · 364 forks observed · 2026-09-03

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Apache DataFusion Comet is a high-performance accelerator plugin for Apache Spark that keeps queries Arrow-native end-to-end, executing operators and expressions as Rust code via the Apache DataFusion engine. It requires no code changes to existing Spark SQL, DataFrame, or PySpark workloads and delivers roughly 2x speedup on TPC-DS benchmarks.

Use cases

• speed up existing Spark SQL jobs without code changes
• reduce cloud costs for large Spark workloads
• accelerate Parquet scans for Apache Iceberg tables
• run TPC-DS style analytics faster on commodity hardware
• replace row-based Spark execution with Arrow columnar processing
• offload Spark operators and shuffles to a native Rust engine

When to choose

• you have existing Spark workloads and want faster queries with no code changes
• you want to cut compute costs by running the same jobs on fewer resources
• your data is in Parquet or Iceberg format on Linux clusters
• you prefer an Apache Software Foundation project with active releases

When to avoid

• you run Spark on macOS in production (published jars bundle Linux native libraries only)
• you rely on proprietary cloud-provider Spark forks
• you need Spark 3.4 or JDK 11 (deprecated)
• your queries use features Comet does not support and frequent fallback negates gains

Facets

library · maturity active

data-science etl streaming big-data analytics databases jvm rust apache-spark apache-arrow datafusion query-acceleration columnar parquet iceberg native-execution query-engine data-engineering linux spark

Member repositories

For agents

markdown · JSON · MCP: product_card(name="apache/datafusion-comet")