jupyter-incubator/sparkmagic

Health v2 · maintenance only

Flags: no_license

Adoption not part of the score

1366 stars · 443 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Sparkmagic is a set of Jupyter magics and kernels for interactively working with remote Spark clusters through a REST server such as Livy, Lighter, or Ilum. It lets users run Spark code in multiple languages, execute SparkSQL queries, visualize results, and exchange data with remote clusters from Jupyter notebooks.

Use cases

• run Spark code against a remote Spark cluster from a Jupyter notebook
• execute SparkSQL queries with the %%sql magic and visualize results
• capture SQL query output as pandas dataframes for use with Python libraries
• connect Jupyter to a Livy server with Basic or Kerberos authentication
• send local files or dataframes to a remote Spark cluster
• manage Spark sessions and view application info and logs from a notebook

When to choose

• you need to work with a remote Spark cluster from Jupyter without a local Spark installation
• you want interactive SparkSQL querying with automatic visualization in notebooks
• your cluster exposes sessions via Livy, Lighter, or Ilum

When to avoid

• you run Spark locally and prefer pyspark directly without a REST intermediary
• you need a non-notebook programmatic Spark client
• your cluster has no Livy-compatible REST server

Facets

library · maturity active

sdk data-science developer-tools data-science big-data developer-tools analytics python cross-platform spark livy jupyter pyspark ipython-magics remote-spark-cluster sql-magic kerberos pandas

Member repositories

For agents

markdown · JSON · MCP: product_card(name="jupyter-incubator/sparkmagic")