databricks/koalas

Health v2 · maintenance only

Adoption not part of the score

3372 stars · 370 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Koalas implements the pandas DataFrame API on top of Apache Spark, letting data scientists use familiar pandas code on distributed big data. It is now deprecated and merged into PySpark as of Apache Spark 3.2.

Use cases

• run pandas code on large distributed datasets with spark
• scale existing pandas dataframes to big data
• use one codebase for pandas and spark
• migrate pandas workflows to spark without rewriting
• do data science on big data with familiar api

When to choose

• you are on Spark 3.1 or below and want the pandas API on Spark
• you need a single codebase that works with both pandas and Spark
• your team knows pandas but not the Spark DataFrame API

When to avoid

• you use Apache Spark 3.2 or above - use PySpark's built-in pandas API instead
• you are starting a new project - the project is deprecated and in maintenance mode
• your datasets fit comfortably in memory on a single machine - plain pandas is simpler

Facets

library · maturity maintenance

data-science etl streaming big-data data-science python cross-platform pandas apache-spark dataframe distributed-computing deprecated pyspark data-engineering

Member repositories

For agents

markdown · JSON · MCP: product_card(name="databricks/koalas")