vaexio/vaex

Health v2 · maintenance only

Adoption not part of the score

8507 stars · 601 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Vaex is a high-performance Python DataFrame library for lazy, out-of-core processing of large tabular datasets, using memory mapping and zero-copy operations to process over a billion rows per second. It includes visualization, statistics on N-dimensional grids, fast groupby/join operations, and a machine-learning subpackage, with a Pandas-like API.

Use cases

• analyze a 100GB CSV file that doesn't fit in RAM
• compute statistics on billions of rows quickly
• visualize and explore large tabular datasets interactively in Jupyter
• convert huge CSV files to HDF5 or Arrow format
• do fast groupby aggregations on big data
• build machine learning pipelines on large datasets on a laptop
• join two large tables without copying data into memory

When to choose

• your tabular data exceeds available memory and you want to avoid setting up a cluster
• you need fast statistics, groupby, or joins on billions of rows on a single machine
• you want interactive visualization of large datasets in Jupyter
• you want a Pandas-like API for out-of-core data processing

When to avoid

• your datasets fit comfortably in memory and Pandas or Polars suffice
• you need distributed computing across a cluster (use Spark or Dask)
• you need heavy ecosystem support or very active development
• you work with unstructured data like text or images rather than tabular data

Facets

library · maturity maintenance

data-science data-visualization machine-learning etl data-science big-data data-visualization machine-learning python cross-platform dataframe out-of-core memory-mapping hdf5 apache-arrow big-data jupyter

Member repositories

For agents

markdown · JSON · MCP: product_card(name="vaexio/vaex")