Open-source big-data software
483 open-source libraries, tools and applications tagged big-data, ranked by how closely big-data is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
483 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| ClickHouse ClickHouse is an open-source column‑oriented OLAP database management system designed for real‑time analytics on massive datasets. It provi… | 96 | 50464 | active |
| Apache Airflow Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows as directed acyclic graphs (… | 98 | 47148 | stable |
| Apache Spark Apache Spark is a unified analytics engine for large-scale data processing that provides high‑level APIs in Scala, Java, Python, and R. It … | 77 | 44159 | active |
| Apache Spark Apache Spark is a unified analytics engine for large-scale data processing, providing high-level APIs in Scala, Java, Python, and R over an… | 77 | 44159 | stable |
| Polars Polars is an extremely fast analytical query engine for DataFrames written in Rust, with multi-threaded vectorized execution, lazy query op… | 92 | 39857 | stable |
| SeaweedFS SeaweedFS is a highly scalable distributed storage system that provides S3‑compatible object storage, POSIX‑like file system access, and Ic… | 95 | 35551 | active |
| RustFS RustFS is a high‑performance, distributed object storage system written in Rust. It provides S3‑compatible APIs and can interoperate with M… | 95 | 35079 | active |
| RustFS RustFS is a high-performance, distributed, S3-compatible object storage system written in Rust, positioned as an Apache 2.0-licensed altern… | 95 | 35079 | active |
| Apache Kafka Apache Kafka is an open-source distributed event streaming platform that provides durable, high‑throughput messaging. It enables real‑time … | 76 | 33918 | active |
| Apache Kafka Apache Kafka is an open-source distributed event streaming platform for building high-performance data pipelines, streaming analytics, and … | 76 | 33918 | stable |
| alibaba/canal Canal is an Alibaba open-source component that parses MySQL binlog to provide incremental data subscription and consumption. It masquerades… | 62 | 29740 | stable |
| Apache Flink Apache Flink is an open‑source distributed processing engine for stateful computations over unbounded and bounded data streams. It provides… | 77 | 26413 | active |
| Apache Flink Apache Flink is an open-source distributed stream processing framework for stateful computations over unbounded and bounded data streams, w… | 77 | 26413 | stable |
| TDengine TDengine is an open‑source, high‑performance time‑series database built for industrial IoT and large‑scale sensor data. It provides built‑i… | 90 | 25149 | active |
| TDengine TDengine is an open-source, cloud-native time-series database (TSDB) written in C, purpose-built for IoT, connected vehicles, industrial Io… | 90 | 25149 | active |
| Apache RocketMQ Apache RocketMQ is a distributed cloud-native messaging and streaming platform built for low-latency, high-throughput, financially reliable… | 88 | 22629 | stable |
| Airbyte Airbyte is an open-source data movement platform providing 600+ connectors for replicating data from APIs, databases, and files into wareho… | 77 | 22234 | active |
| Dgraph Dgraph is a horizontally scalable, distributed graph database that offers a native GraphQL API and ACID transactions. It is designed for re… | 97 | 21804 | active |
| Apache ShardingSphere Apache ShardingSphere is an open‑source distributed database ecosystem that provides sharding, read/write splitting, encryption, and distri… | 77 | 20808 | active |
| Apache ShardingSphere Apache ShardingSphere is an enterprise distributed database ecosystem that turns heterogeneous databases (MySQL, PostgreSQL, etc.) into a d… | 77 | 20808 | stable |
| spotify/luigi Luigi is a Python package for building complex pipelines of long-running batch jobs. It handles dependency resolution, workflow management,… | 84 | 18780 | stable |
| alibaba/DataX DataX is Alibaba's open-source offline data synchronization framework, the open version of Alibaba Cloud DataWorks data integration. It syn… | 60 | 17364 | stable |
| apache/arrow Apache Arrow is a language-independent columnar in-memory data format specification plus multi-language libraries (C++, Python/PyArrow, Jav… | 95 | 17212 | stable |
| Presto Presto is a distributed SQL query engine for running interactive queries against large datasets across heterogeneous data sources such as H… | 97 | 16757 | stable |
| dagster-io/dagster Dagster is an open-source Python data orchestration platform for developing, producing, and observing data assets, with integrated lineage,… | 95 | 16287 | active |
| Apache Doris Apache Doris is an open-source MPP-based real-time analytical database that delivers sub-second queries over massive datasets, combining a … | 98 | 16078 | stable |
| scylladb/scylladb ScyllaDB is an open-source, high-performance NoSQL wide-column data store built in C++ on the shared-nothing Seastar framework. It is API-c… | 76 | 15784 | stable |
| apache/hadoop Apache Hadoop is an open-source framework for reliable, scalable distributed storage (HDFS) and processing (MapReduce, YARN) of large data … | 77 | 15694 | stable |
| Apache Pulsar Apache Pulsar is a cloud-native, distributed pub-sub messaging and streaming platform originally developed at Yahoo and now a top-level Apa… | 99 | 15346 | stable |
| elastic/logstash Logstash is an open-source server-side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and … | 95 | 14966 | stable |
| JuiceFS JuiceFS is a high-performance, POSIX-compatible distributed file system that stores file data in object storage (e.g., Amazon S3) and metad… | 96 | 14549 | stable |
| Apache DolphinScheduler Apache DolphinScheduler is a modern data orchestration platform for building high-performance workflows with low-code drag-and-drop tooling… | 92 | 14510 | stable |
| apache/druid Apache Druid is a high-performance, distributed, real-time analytics database written in Java for fast OLAP-style slice-and-dice queries on… | 95 | 14089 | stable |
| Dask Dask is a flexible parallel and distributed computing library for Python that scales pandas, NumPy, scikit-learn, and other PyData tools to… | 96 | 13934 | stable |
| opensearch-project/OpenSearch OpenSearch is an open-source, distributed, RESTful search and analytics suite for full-text search, log analytics, application monitoring, … | 94 | 13813 | stable |
| trinodb/trino Trino is a fast, distributed ANSI SQL query engine for big data analytics, formerly known as PrestoSQL. It queries data in place across div… | 95 | 13345 | stable |
| Debezium Debezium is an open source distributed platform for change data capture (CDC) that streams row-level database changes as events, most commo… | 77 | 13237 | stable |
| Citus Citus is an open-source PostgreSQL extension (written in C) that transforms Postgres into a distributed database by sharding tables across … | 95 | 12803 | active |
| redpanda-data/redpanda Redpanda is a Kafka API-compatible streaming data platform written in C++ on the Seastar framework, with no ZooKeeper or JVM dependency. It… | 88 | 12571 | stable |
| Nebula Graph NebulaGraph is a distributed, open-source graph database written in C++ that handles large graph datasets with millisecond latency and hori… | 65 | 12409 | active |
| provectus/kafka-ui UI for Apache Kafka is a free, open-source web UI for monitoring and managing Apache Kafka clusters. It provides a lightweight dashboard to… | 23 | 12297 | active |
| StarRocks StarRocks is a high-performance distributed OLAP database and query engine built on an MPP architecture with a fully vectorized execution e… | 94 | 12174 | active |
| Manticore Search Manticore Search is an open-source search database written in C++ that provides fast full-text, vector, and hybrid search with real-time in… | 98 | 12064 | stable |
| quickwit-oss/quickwit Quickwit is an open-source, cloud-native search engine written in Rust and powered by Tantivy, designed to run sub-second search and analyt… | 93 | 11676 | active |
| AutoMQ/automq AutoMQ is a cloud-native, 100% Apache Kafka-compatible streaming platform that replaces broker-local disks with S3-compatible object storag… | 94 | 10909 | active |
| modin-project/modin Modin is a drop-in replacement for pandas that scales DataFrame operations across all CPU cores using Ray, Dask, or Unidist as execution en… | 61 | 10389 | active |
| oceanbase/oceanbase OceanBase is a distributed relational database engine developed by Ant Group, offering MySQL compatibility, Paxos-based high availability, … | 98 | 10300 | stable |
| apache/cassandra Apache Cassandra is an open-source NoSQL distributed wide-column database written in Java, designed for linear scalability and fault tolera… | 76 | 10115 | stable |
| NVIDIA/cudf cuDF is a GPU-accelerated DataFrame library for tabular data processing, part of NVIDIA's RAPIDS suite. It provides a pandas-compatible Pyt… | 95 | 9780 | stable |
| Apache SeaTunnel Apache SeaTunnel is a distributed, high-performance data integration platform for synchronizing massive amounts of data across hundreds of … | 90 | 9700 | stable |
| apache/datafusion Apache DataFusion is an extensible query engine written in Rust that uses Apache Arrow as its in-memory columnar format. It provides SQL an… | 77 | 9505 | active |
| databendlabs/databend Databend is an open-source, cloud-native data warehouse built in Rust that runs entirely on object storage (S3, Azure, GCS). It unifies BI … | 90 | 9471 | active |
| apache/iceberg Apache Iceberg is a high-performance open table format for huge analytic datasets, bringing SQL table reliability to big data lakes. This r… | 94 | 9322 | stable |
| Delta Lake Delta Lake is an open-source storage framework and table format that brings ACID transactions, scalable metadata handling, schema enforceme… | 95 | 9053 | stable |
| redpanda-data/connect Redpanda Connect (formerly Benthos) is a declarative stream processor that moves data between hundreds of sources and sinks with transforma… | 94 | 8778 | active |
| apache/beam Apache Beam is an open-source unified programming model and SDK set (Java, Python, Go, SQL, TypeScript) for defining batch and streaming da… | 95 | 8688 | stable |
| pentaho/pentaho-kettle Pentaho Data Integration (Kettle) is an open-source ETL application for designing and running data extraction, transformation, and loading … | 66 | 8400 | active |
| arkime/arkime Arkime is an open-source, large-scale network analysis, full packet capture, and session indexing system that stores traffic in standard PC… | 99 | 7536 | active |
| H2O H2O-3 is an open-source, distributed, in-memory machine learning platform implementing algorithms such as GLM, GBM/XGBoost, Random Forest, … | 75 | 7509 | stable |
| Feast Feast is an open-source feature store for machine learning that manages offline stores for historical training data and low-latency online … | 99 | 7327 | stable |
| Alluxio/alluxio Alluxio is an open-source distributed caching and data orchestration platform that sits between compute frameworks (Spark, Presto, Trino, P… | 64 | 7248 | stable |
| didi/KnowStreaming Know Streaming is a cloud-native Kafka management and control platform built from years of operational experience at Didi. It provides zero… | 81 | 7174 | active |
| Vespa Vespa is an open-source, distributed AI search platform and serving engine that combines full-text search, vector/tensor search, and machin… | 93 | 7117 | stable |
| Snowplow Snowplow is a Customer Data Infrastructure platform that collects, validates, enriches, and streams event-level behavioral data from web, m… | 59 | 7034 | stable |
| apache/zeppelin Apache Zeppelin is a web-based notebook for interactive, data-driven analytics and collaborative documents. It supports SQL, Scala, Python,… | 76 | 6663 | stable |
| Hazelcast Hazelcast is a unified real-time data platform combining distributed stream processing with a fast, in-memory data store. It lets applicati… | 80 | 6622 | stable |
| apache/flink-cdc Flink CDC is a distributed streaming data integration tool built on Apache Flink that captures change data from databases like MySQL and Po… | 79 | 6482 | active |
| Apache Hudi Apache Hudi is an open data lakehouse platform built on a high-performance open table format that brings database functionality like transa… | 96 | 6336 | stable |
| Pachyderm Pachyderm is a data-centric pipeline platform that automates data transformations with built-in data versioning and lineage tracking. It ru… | 30 | 6309 | active |
| apache/nifi Apache NiFi is an easy-to-use, powerful, and reliable system to process and distribute data, built on the JVM. It provides a browser-based … | 93 | 6249 | stable |
| apache/pinot Apache Pinot is an open-source distributed OLAP datastore purpose-built for low-latency, high-throughput real-time analytics. It ingests da… | 82 | 6163 | stable |
| OpenAtomFoundation/pikiwidb PikiwiDB (Pika) is a Redis-compatible, persistent key-value database built on RocksDB, developed by Qihoo's infrastructure team. It speaks … | 87 | 6130 | active |
| apache/hive Apache Hive is a distributed, fault-tolerant data warehouse system that enables reading, writing, and managing petabytes of data in distrib… | 77 | 6030 | active |
| DataX Web DataX-Web is a distributed data synchronization tool built on top of Alibaba's DataX, providing a web UI to visually configure and manage d… | 23 | 6016 | stable |
| data load tool (dlt) dlt (data load tool) is an open-source Python library for building ELT data pipelines that extract data from REST APIs, SQL databases, clou… | 99 | 5955 | stable |
| JanusGraph JanusGraph is an open-source, distributed graph database optimized for storing and querying graphs with billions of vertices and edges acro… | 79 | 5857 | active |
| Eventual-Inc/Daft Daft is a high-performance distributed data engine with a Python dataframe API, implemented in Rust, designed for AI and multimodal workloa… | 95 | 5804 | active |
| Nuclio Nuclio is a high-performance open-source serverless (FaaS) platform for real-time event and data processing, deployable standalone via Dock… | 92 | 5760 | stable |
| cubefs/cubefs CubeFS is a CNCF-graduated, cloud-native distributed file and object storage system written in Go. It supports S3, HDFS, and POSIX access p… | 87 | 5674 | stable |
| apache/hbase Apache HBase is an open-source, distributed, versioned, column-oriented NoSQL database modeled after Google Bigtable, running on top of Apa… | 91 | 5560 | stable |
| lakeFS lakeFS is an open-source data version control service that turns object storage (S3, Azure Blob, GCS) into a Git-like repository with branc… | 93 | 5547 | stable |
| fluvio-community/fluvio Fluvio is a distributed data streaming engine written in Rust that combines a Kafka-like event streaming platform with the Stateful DataFlo… | 79 | 5256 | active |
| microsoft/SynapseML SynapseML (formerly MMLSpark) is an open-source machine learning library built on Apache Spark that provides simple, composable, distribute… | 84 | 5247 | active |
| apache/calcite Apache Calcite is a dynamic data management framework for the JVM that supplies the building blocks of a database—an industry-standard SQL … | 77 | 5194 | stable |
| Jitsu Jitsu is an open-source, self-hostable event data platform and Segment alternative that collects event data from websites, apps, and server… | 95 | 5108 | active |
| Apache Ignite Apache Ignite is a distributed, memory-first database with in-memory speed, ACID transactions, and SQL support across a cluster. It can ser… | 76 | 5082 | stable |
| ArroyoSystems/arroyo Arroyo is a distributed stream processing engine written in Rust that lets users run stateful computations on high-volume real-time data st… | 76 | 5048 | active |
| microsoft/SPTAG SPTAG is a C++ library from Microsoft for large-scale approximate nearest neighbor (ANN) vector search, offering kd-tree/balanced k-means t… | 75 | 5019 | active |
| deepseek-ai/smallpond Smallpond is a lightweight distributed data processing framework built on DuckDB and DeepSeek's 3FS shared file system. It lets users proce… | 21 | 5017 | active |
| Apache AGE Apache AGE is a PostgreSQL extension that adds graph database capabilities on top of existing relational databases, supporting openCypher q… | 88 | 4873 | active |
| Amundsen Amundsen is an open-source data discovery and metadata engine that indexes data resources such as tables, dashboards, and streams, and powe… | 10 | 4782 | active |
| YDB YDB is an open-source distributed SQL DBMS written in C++ that combines horizontal scalability, high availability, and strict consistency w… | 98 | 4781 | stable |
| apache/rocketmq-externals The Apache RocketMQ externals repository is the community home for incubating ecosystem projects around the RocketMQ message broker, such a… | 74 | 4593 | active |
| RudderStack RudderStack's open-source event streaming server, a privacy- and security-focused Segment alternative written in Go. It collects customer e… | 96 | 4504 | active |
| rom1504/img2dataset A Python tool that downloads large sets of image URLs and packages them into machine learning datasets, with resizing and caption support. … | 52 | 4454 | active |
| CrateDB CrateDB is a distributed, horizontally scalable SQL database built on Lucene, designed for real-time analytics on massive datasets. It supp… | 95 | 4444 | active |
| Apache StreamPark Apache StreamPark is a streaming application development framework and one-stop cloud-native real-time computing platform for Apache Flink … | 72 | 4339 | stable |
| zendesk/maxwell Maxwell's Daemon is a change data capture (CDC) application that reads MySQL binlogs and emits row-level changes as JSON to Kafka, Kinesis,… | 93 | 4259 | active |
| facebookincubator/velox Velox is a composable, extensible C++ execution engine library for data management systems, created by Meta. It provides reusable vectorize… | 77 | 4233 | active |
| pydata/xarray Xarray is a Python library that adds labeled dimensions, coordinates, and attributes on top of NumPy-like N-dimensional arrays and datasets… | 98 | 4207 | stable |
page 1 / 5 next →