Dask

Health v2 · maintenance only

Adoption not part of the score

13934 stars · 1976 forks observed · 2026-10-11

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Dask is a flexible parallel and distributed computing library for Python that scales pandas, NumPy, scikit-learn, and other PyData tools to larger-than-memory and multi-machine workloads. It provides task scheduling with arbitrary dependencies, parallel dataframes and arrays, and a distributed scheduler deployable from a laptop to clusters on cloud or Kubernetes.

Use cases

• process a 100 GB CSV that doesn't fit in RAM with pandas-like code
• parallelize a for loop that processes thousands of files
• analyze terabytes of multi-dimensional array data with xarray
• scale machine learning training and hyperparameter optimization across a cluster
• run distributed computations on Kubernetes or cloud infrastructure
• speed up groupby and merge operations on large parquet datasets

When to choose

• you already use pandas/NumPy and hit memory or performance limits
• you need flexible arbitrary task graphs, not just data-parallel jobs
• you want a lighter, more Pythonic alternative to Spark
• you need to scale from a laptop to a cluster with the same API

When to avoid

• your datasets fit comfortably in memory and pandas is fast enough
• you need heavy SQL-based ETL where a query engine or Spark is better suited
• you need strict low-latency streaming rather than batch/task computation
• your team requires a JVM-based ecosystem with mature enterprise tooling

Facets

library · maturity stable

concurrency data-science etl machine-learning streaming developer-tools data-science big-data analytics machine-learning microservices python cross-platform cloud parallel-computing task-scheduling dataframes numpy pandas distributed-computing pydata larger-than-memory data-engineering kubernetes

Member repositories

For agents

markdown · JSON · MCP: product_card(name="dask/dask")