# databricks/spark-deep-learning

Deep Learning Pipelines for Apache Spark

Repository: https://github.com/databricks/spark-deep-learning
Canonical: https://ross.abutalabs.com/products/spark-deep-learning
Homepage: https://databricks.github.io/spark-deep-learning
Language: Python
License: Apache-2.0
License Family: permissive
Last push: 2023-03-30T20:19:05+00:00

## Health v2 (maintenance only)
Score: 23/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 8, longevity 100
- inputs: {"age_days": 3381, "days_push": 1252, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1987, forks 489 (observed 2026-08-28T04:06:02.874434+00:00)

## What it is
Deep Learning Pipelines for Apache Spark, now reduced to the HorovodRunner component for distributed deep learning training via Horovod on Spark clusters. The full Spark ML pipelines API lives only in previous releases, and distributed execution requires Databricks Runtime for Machine Learning.

## Use cases
- run distributed deep learning training on a Spark cluster
- train TensorFlow or PyTorch models with Horovod on Databricks
- scale GPU model training across Spark worker nodes
- launch Horovod jobs as Spark jobs with cluster setup managed for me
- test Horovod training locally in a single Python process

## When to avoid
- you need the old Spark Deep Learning Pipelines API (KerasImageFileTransformer, etc.) - only available in archived releases
- you want a fully open-source distributed training solution without Databricks - the OSS version only runs locally
- you are not using Spark or Databricks at all
- you need actively developed features - the repo is mostly in maintenance

## Facets
- artifact type: library
- maturity: maintenance
- function: machine-learning, deep-learning, llm-training
- domain: deep-learning, machine-learning, big-data, microservices
- platform: python, jvm, cloud
- tags: apache-spark, horovod, distributed-training, databricks, gpu, docker

## Member repositories
- databricks/spark-deep-learning (main) score 23

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:06:02.874434+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:03:00.234246+00:00, confidence not recorded.
  - readme: https://github.com/databricks/spark-deep-learning (fetched 2026-08-28T04:06:02.874434+00:00, sha d6a9f3ac8321)
  - homepage: https://databricks.github.io/spark-deep-learning (fetched 2026-08-29T10:42:46.550584+00:00, sha 87dfab9661af)
- Data as of 2026-08-30T08:39:29.467469+00:00.