Apache Spark
Apache Spark - A unified analytics engine for large-scale data processing observed · 2026-10-11
Health v2 · maintenance only
77/100
- Activity 99
- Release rhythm 35
- Longevity 100
Flags: no_releases
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-10-21. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 4620
- days_rel: n/a
- days_push: 10
- n_releases_24m: 0
Adoption not part of the score
44159 stars · 29425 forks observed · 2026-10-11
What it is AI-extracted, prompt v1, taxonomy v1, 2026-09-28, confidence 0.98
Apache Spark is a unified analytics engine for large-scale data processing that provides high‑level APIs in Scala, Java, Python, and R. It includes libraries for SQL, streaming, machine learning, and graph processing.
Use cases
- run batch ETL pipelines on petabyte‑scale datasets
- perform interactive SQL queries on large data lakes
- process real‑time event streams with low latency
- train machine‑learning models on distributed clusters
- execute graph analytics on massive networks
When to choose
- you need a single engine that can handle batch, streaming, and interactive analytics
- your workloads require scaling to thousands of nodes or petabyte‑scale data
- you want to write jobs in multiple languages (Python, Scala, Java, R)
- you need built‑in libraries for SQL, ML, and graph processing
When to avoid
- your data fits comfortably in memory on a single machine and you prefer a lightweight library
- you require ultra‑low‑latency processing below millisecond response times
- you need a pure in‑process solution without a separate cluster manager
Facets
framework · maturity active
etl streaming big-data data-science machine-learning analytics windows jvm python cloud distributed-computing cluster-management spark-sql mlib structured-streaming sql linux macos docker kubernetes
9 sources
- readme: https://github.com/apache/spark · fetched 2026-10-11 · 70f6be2e8ea1
- homepage: https://spark.apache.org/ · fetched 2026-09-14 · a291a9d9a452
- site_page: https://spark.apache.org/docs/latest · fetched 2026-09-14 · 07bd3c2857f4
- site_page: https://spark.apache.org/documentation.html · fetched 2026-09-14 · 731c16e1c2df
- site_page: https://spark.apache.org/docs/latest/quick-start.html · fetched 2026-09-14 · e471ff564584
- site_page: https://spark.apache.org/docs/latest/sql-performance-tuning.html · fetched 2026-09-14 · ad4596579581
- site_page: https://spark.apache.org/docs/latest/sql-ref-ansi-compliance.html · fetched 2026-09-14 · 647aec5b39fa
- site_page: https://spark.apache.org/docs/latest/sql-data-sources-json.html · fetched 2026-09-14 · dde447752a94
- site_page: https://spark.apache.org/faq.html · fetched 2026-09-14 · 707d496b9598
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| apache/spark | main | 77 |
| apache/spark-website | docs | 74 |
For agents
Data as of 2026-10-19T08:05:51.212177+00:00 · Report a problem