# databricks/spark-sklearn

(Deprecated) Scikit-learn integration package for Apache Spark

Repository: https://github.com/databricks/spark-sklearn
Canonical: https://ross.abutalabs.com/products/spark-sklearn
Language: Python
License: Apache-2.0
License Family: permissive
Topics: apache-spark, scikit-learn, grid-search, parameter-tuning, machine-learning
Archived: true
Last push: 2019-12-03T18:37:45+00:00

## Health v2 (maintenance only)
Score: 10/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 8, longevity 100
- inputs: {"age_days": 4018, "days_push": 2465, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: archived
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1071, forks 224 (observed 2026-08-28T04:03:28.316183+00:00)

## What it is
A deprecated Python library from Databricks that integrates scikit-learn with Apache Spark, primarily to distribute grid-search cross-validation and model training tasks across a Spark cluster. It also converts Spark DataFrames into numpy arrays or sparse matrices.

## Use cases
- distribute scikit-learn GridSearchCV across a Spark cluster
- train and evaluate multiple scikit-learn models in parallel
- convert Spark DataFrames to numpy arrays or sparse matrices
- run hyperparameter tuning on small datasets using Spark
- distribute scipy sparse matrices as a Spark dataset

## When to choose
- you are maintaining legacy code already built on spark-sklearn
- you need a quick way to parallelize scikit-learn grid search on an existing Spark cluster with older library versions

## When to avoid
- starting a new project - use joblib-spark (joblibspark) instead as officially recommended
- you need to distribute individual learning algorithms rather than simple tasks like grid search
- your datasets do not fit in memory - use Spark MLlib instead
- you use scikit-learn >= 0.20 or recent Spark versions, which are incompatible

## Facets
- artifact type: library
- maturity: abandoned
- function: machine-learning, concurrency
- domain: machine-learning, big-data, data-science
- platform: python, cross-platform
- tags: apache-spark, scikit-learn, grid-search, hyperparameter-tuning, deprecated

## Member repositories
- databricks/spark-sklearn (main) score 10

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:03:28.316183+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:53:48.729250+00:00, confidence not recorded.
  - readme: https://github.com/databricks/spark-sklearn (fetched 2026-08-28T04:03:28.316183+00:00, sha cd0e16f9c9ce)
  - registry_pypi: https://pypi.org/pypi/spark-sklearn/json (fetched 2026-08-29T12:56:12.707336+00:00, sha c6db87ec8ddc)
- Data as of 2026-08-30T08:39:29.467469+00:00.