# jupyter-incubator/sparkmagic

Jupyter magics and kernels for working with remote Spark clusters

Repository: https://github.com/jupyter-incubator/sparkmagic
Canonical: https://ross.abutalabs.com/products/sparkmagic
Language: Python
License: NOASSERTION
License Family: other
Topics: spark, kernel, cluster, livy, magic, sql-query, pandas-dataframe, jupyter, pyspark, kerberos, notebook, jupyter-notebook
Last push: 2025-09-09T14:08:11+00:00

## Health v2 (maintenance only)
Score: 41/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 41, release rhythm 8, longevity 100
- inputs: {"age_days": 3999, "days_push": 358, "days_rel": 422, "gap_med": 236, "n_releases_24m": 2}
- flags: no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1366, forks 443 (observed 2026-08-28T04:04:31.329279+00:00)

## What it is
Sparkmagic is a set of Jupyter magics and kernels for interactively working with remote Spark clusters through a REST server such as Livy, Lighter, or Ilum. It lets users run Spark code in multiple languages, execute SparkSQL queries, visualize results, and exchange data with remote clusters from Jupyter notebooks.

## Use cases
- run Spark code against a remote Spark cluster from a Jupyter notebook
- execute SparkSQL queries with the %%sql magic and visualize results
- capture SQL query output as pandas dataframes for use with Python libraries
- connect Jupyter to a Livy server with Basic or Kerberos authentication
- send local files or dataframes to a remote Spark cluster
- manage Spark sessions and view application info and logs from a notebook

## When to choose
- you need to work with a remote Spark cluster from Jupyter without a local Spark installation
- you want interactive SparkSQL querying with automatic visualization in notebooks
- your cluster exposes sessions via Livy, Lighter, or Ilum

## When to avoid
- you run Spark locally and prefer pyspark directly without a REST intermediary
- you need a non-notebook programmatic Spark client
- your cluster has no Livy-compatible REST server

## Facets
- artifact type: library
- maturity: active
- function: sdk, data-science, developer-tools
- domain: data-science, big-data, developer-tools, analytics
- platform: python, cross-platform
- tags: spark, livy, jupyter, pyspark, ipython-magics, remote-spark-cluster, sql-magic, kerberos, pandas

## Member repositories
- jupyter-incubator/sparkmagic (main) score 41

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:31.329279+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T04:41:13.101173+00:00, confidence not recorded.
  - readme: https://github.com/jupyter-incubator/sparkmagic (fetched 2026-08-28T04:04:31.329279+00:00, sha d068a024ed48)
  - registry_pypi: https://pypi.org/pypi/sparkmagic/json (fetched 2026-08-29T11:58:22.493494+00:00, sha 92bd7f8b220a)
- Data as of 2026-08-30T08:39:29.467469+00:00.