# apache/datafusion-comet

Apache DataFusion Comet Spark Accelerator

Repository: https://github.com/apache/datafusion-comet
Canonical: https://ross.abutalabs.com/products/datafusion-comet
Homepage: https://datafusion.apache.org/comet
Language: Scala
License: Apache-2.0
License Family: permissive
Topics: arrow, datafusion, rust, spark
Last push: 2026-09-02T20:46:17+00:00

## Health v2 (maintenance only)
Score: 81/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 100, release rhythm 64, longevity 68
- inputs: {"age_days": 961, "days_push": 0, "days_rel": 26, "gap_med": null, "n_releases_24m": 1}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1262, forks 364 (observed 2026-09-03T02:15:09.172587+00:00)

## What it is
Apache DataFusion Comet is a high-performance accelerator plugin for Apache Spark that keeps queries Arrow-native end-to-end, executing operators and expressions as Rust code via the Apache DataFusion engine. It requires no code changes to existing Spark SQL, DataFrame, or PySpark workloads and delivers roughly 2x speedup on TPC-DS benchmarks.

## Use cases
- speed up existing Spark SQL jobs without code changes
- reduce cloud costs for large Spark workloads
- accelerate Parquet scans for Apache Iceberg tables
- run TPC-DS style analytics faster on commodity hardware
- replace row-based Spark execution with Arrow columnar processing
- offload Spark operators and shuffles to a native Rust engine

## When to choose
- you have existing Spark workloads and want faster queries with no code changes
- you want to cut compute costs by running the same jobs on fewer resources
- your data is in Parquet or Iceberg format on Linux clusters
- you prefer an Apache Software Foundation project with active releases

## When to avoid
- you run Spark on macOS in production (published jars bundle Linux native libraries only)
- you rely on proprietary cloud-provider Spark forks
- you need Spark 3.4 or JDK 11 (deprecated)
- your queries use features Comet does not support and frequent fallback negates gains

## Facets
- artifact type: library
- maturity: active
- function: data-science, etl, streaming
- domain: big-data, analytics, databases
- platform: jvm, rust
- tags: apache-spark, apache-arrow, datafusion, query-acceleration, columnar, parquet, iceberg, native-execution, query-engine, data-engineering, linux, spark

## Member repositories
- apache/datafusion-comet (main) score 81

## Provenance
- Observed fields: from GitHub, fetched 2026-09-03T02:15:09.172587+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T05:05:34.231132+00:00, confidence not recorded.
  - readme: https://github.com/apache/datafusion-comet (fetched 2026-09-03T02:15:09.172587+00:00, sha 41154ac1324f)
  - homepage: https://datafusion.apache.org/comet (fetched 2026-08-29T12:17:10.356415+00:00, sha 42f8c9b44a17)
  - site_page: https://datafusion.apache.org/comet/about/index.html (fetched 2026-08-29T12:17:10.361049+00:00, sha f79bba2bc6c3)
  - site_page: https://datafusion.apache.org/comet/user-guide/latest/installation.html (fetched 2026-08-29T12:17:10.362692+00:00, sha 8dadd8ba0a33)
  - site_page: https://datafusion.apache.org/comet/changelog/index.html (fetched 2026-08-29T12:17:10.359219+00:00, sha cd9e11aa04e6)
- Data as of 2026-08-30T08:39:29.467469+00:00.