apache/spark
Apache Spark - A unified analytics engine for large-scale data processing observed · 2026-08-28
Health v2 · maintenance only
77/100
- Activity 99
- Release rhythm 35
- Longevity 100
Flags: no_releases
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 4572
- days_rel: n/a
- days_push: 7
- n_releases_24m: 0
Adoption not part of the score
43882 stars · 29348 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
Apache Spark is a unified analytics engine for large-scale data processing, providing high-level APIs in Scala, Java, Python, and R over an optimized distributed execution engine. It bundles higher-level tools including Spark SQL and DataFrames, a pandas API, MLlib for machine learning, GraphX for graph processing, and Structured Streaming for real-time stream processing.
Use cases
- process petabyte-scale datasets across a cluster
- run distributed SQL queries for analytics and reporting
- build ETL pipelines over batch and streaming data
- train machine learning models at scale with MLlib
- analyze large JSON, CSV, or Parquet files with DataFrames
- perform exploratory data analysis on big data
- process real-time event streams with Structured Streaming
- run graph analytics on large graphs with GraphX
When to choose
- you need to process data too large for a single machine
- you want one engine for batch, streaming, SQL, and ML workloads
- you need a mature, battle-tested big data framework with a large ecosystem
- your team works in Python, Scala, Java, or SQL and wants pandas-like or SQL interfaces
- you need to query data in HDFS, S3, Hive, Cassandra, or JDBC databases
When to avoid
- your datasets fit comfortably in memory on one machine - pandas or DuckDB are simpler
- you need low-latency per-event stream processing rather than micro-batch
- you want lightweight ad-hoc analytics without cluster infrastructure overhead
- you need a small-footprint embedded database rather than a distributed engine
Facets
framework · maturity stable
etl streaming machine-learning data-science search-engine big-data data-science machine-learning analytics microservices jvm python cross-platform cloud windows spark pyspark dataframes spark-sql mllib structured-streaming graphx distributed-computing apache scala sql data-engineering docker kubernetes linux macos
9 sources
- readme: https://github.com/apache/spark · fetched 2026-08-28 · 70f6be2e8ea1
- homepage: https://spark.apache.org/ · fetched 2026-08-29 · a291a9d9a452
- site_page: https://spark.apache.org/docs/latest · fetched 2026-08-29 · 07bd3c2857f4
- site_page: https://spark.apache.org/documentation.html · fetched 2026-08-29 · 731c16e1c2df
- site_page: https://spark.apache.org/docs/latest/quick-start.html · fetched 2026-08-29 · e471ff564584
- site_page: https://spark.apache.org/docs/latest/sql-performance-tuning.html · fetched 2026-08-29 · ad4596579581
- site_page: https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html · fetched 2026-08-29 · 647aec5b39fa
- site_page: https://spark.apache.org/docs/latest/sql-data-sources-json.html · fetched 2026-08-29 · dde447752a94
- site_page: https://spark.apache.org/faq.html · fetched 2026-08-29 · 707d496b9598
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| apache/spark | main | 77 |
For agents
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem