databricks/spark-sklearn

Health v2 · maintenance only

Flags: archived

Adoption not part of the score

1071 stars · 224 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A deprecated Python library from Databricks that integrates scikit-learn with Apache Spark, primarily to distribute grid-search cross-validation and model training tasks across a Spark cluster. It also converts Spark DataFrames into numpy arrays or sparse matrices.

Use cases

• distribute scikit-learn GridSearchCV across a Spark cluster
• train and evaluate multiple scikit-learn models in parallel
• convert Spark DataFrames to numpy arrays or sparse matrices
• run hyperparameter tuning on small datasets using Spark
• distribute scipy sparse matrices as a Spark dataset

When to choose

• you are maintaining legacy code already built on spark-sklearn
• you need a quick way to parallelize scikit-learn grid search on an existing Spark cluster with older library versions

When to avoid

• starting a new project - use joblib-spark (joblibspark) instead as officially recommended
• you need to distribute individual learning algorithms rather than simple tasks like grid search
• your datasets do not fit in memory - use Spark MLlib instead
• you use scikit-learn >= 0.20 or recent Spark versions, which are incompatible

Facets

library · maturity abandoned

machine-learning concurrency machine-learning big-data data-science python cross-platform apache-spark scikit-learn grid-search hyperparameter-tuning deprecated

Member repositories

For agents

markdown · JSON · MCP: product_card(name="databricks/spark-sklearn")