databricks/spark-deep-learning

Health v2 · maintenance only

Adoption not part of the score

1987 stars · 489 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Deep Learning Pipelines for Apache Spark, now reduced to the HorovodRunner component for distributed deep learning training via Horovod on Spark clusters. The full Spark ML pipelines API lives only in previous releases, and distributed execution requires Databricks Runtime for Machine Learning.

Use cases

• run distributed deep learning training on a Spark cluster
• train TensorFlow or PyTorch models with Horovod on Databricks
• scale GPU model training across Spark worker nodes
• launch Horovod jobs as Spark jobs with cluster setup managed for me
• test Horovod training locally in a single Python process

When to avoid

• you need the old Spark Deep Learning Pipelines API (KerasImageFileTransformer, etc.) - only available in archived releases
• you want a fully open-source distributed training solution without Databricks - the OSS version only runs locally
• you are not using Spark or Databricks at all
• you need actively developed features - the repo is mostly in maintenance

Facets

library · maturity maintenance

machine-learning deep-learning llm-training deep-learning machine-learning big-data microservices python jvm cloud apache-spark horovod distributed-training databricks gpu docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="databricks/spark-deep-learning")