Open-source big-data software
483 open-source libraries, tools and applications tagged big-data, ranked by how closely big-data is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
483 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| FeatureBaseDB/featurebase FeatureBase (formerly Pilosa) is a distributed analytical database built entirely on bitmap indexes, offering low-latency SQL queries over … | 10 | 2523 | abandoned |
| ByConity/ByConity ByConity is an open-source cloud-native distributed SQL data warehouse built on the ClickHouse 21.8 codebase, developed by ByteDance/Volcan… | 10 | 2237 | abandoned |
| mozilla/MozDef MozDef is Mozilla's Enterprise Defense Platform, an open-source SIEM-style application for automating security incident handling and enabli… | 10 | 2159 | abandoned |
| twitter/summingbird Summingbird is a Scala library for writing MapReduce aggregation programs that look like native collection transformations and run on distr… | 10 | 2121 | abandoned |
| Onyx Onyx is a masterless, fault-tolerant distributed computation system written in pure Clojure that supports both batch and stream processing … | 10 | 2050 | abandoned |
| Tencent/TubeMQ TubeMQ was Tencent's high-performance message queue, donated to the Apache Software Foundation in 2019 and renamed Apache InLong. This repo… | 23 | 1995 | abandoned |
| facebookarchive/LogDevice LogDevice is a scalable, fault-tolerant distributed log storage system written in C++ by Facebook, storing append-only record streams repli… | 10 | 1900 | abandoned |
| gchq/Gaffer Gaffer is a large-scale graph database framework for storing very large graphs with rich properties, supporting backends like Accumulo, HBa… | 10 | 1787 | abandoned |
| discoproject/disco Disco is an open-source distributed MapReduce framework written in Erlang with a Python job API, originally developed at Nokia Research Cen… | 32 | 1628 | abandoned |
| mongodb/mongo-hadoop A Java library that lets MongoDB (or BSON backup files) serve as an input source or output destination for Hadoop MapReduce jobs, with inte… | 10 | 1552 | abandoned |
| alibaba/mdrill Mdrill is an open-source distributed OLAP (online analytical processing) engine from Alibaba's AdMom team, built in Java on top of JStorm/H… | 10 | 1542 | abandoned |
| tensorbase/tensorbase TensorBase is a Rust-based open-source OLAP data warehouse designed as a ClickHouse-compatible, high-performance alternative for big data a… | 23 | 1462 | abandoned |
| mesos/spark This is the original UC Berkeley AMPLab repository for Apache Spark, a fast cluster computing system supporting Java, Scala, and Python. Th… | 32 | 1419 | abandoned |
| yahoo/CaffeOnSpark CaffeOnSpark is a Spark package that brings the Caffe deep learning framework to Hadoop and Spark clusters, enabling distributed neural net… | 10 | 1262 | abandoned |
| uber-archive/AthenaX AthenaX is a SQL-based streaming analytics platform open sourced by Uber, built on Apache Flink and Apache Calcite. It lets users run produ… | 10 | 1222 | abandoned |
| foolcage/fooltrader fooltrader is a Python quantitative analysis and trading framework that crawls, cleans, and structures market data (stocks, futures, forex,… | 23 | 1198 | abandoned |
| eventql/eventql EventQL is a distributed, columnar SQL database written in C++ for large-scale data collection and analytics, supporting streaming writes, … | 23 | 1178 | abandoned |
| killrweather/killrweather KillrWeather is a Scala reference application demonstrating integration of Apache Spark Streaming, Apache Kafka, Apache Cassandra, and Akka… | 32 | 1177 | abandoned |
| twitter-archive/ambrose Twitter Ambrose is a platform for visualization and real-time monitoring of MapReduce data workflows on Hadoop clusters. It provides a web … | 10 | 1169 | abandoned |
| antonmks/Alenka Alenka is a GPU-based column-store database engine written in CUDA that leverages vectorized processing and GPU memory bandwidth for fast a… | 32 | 1166 | abandoned |
| Spring Cloud Data Flow Spring Cloud Data Flow is a microservices-based toolkit for orchestrating streaming and batch data processing pipelines built from Spring B… | 10 | 1140 | abandoned |
| kubernetes-retired/kube-batch kube-batch is a batch scheduler for Kubernetes designed for high-performance workloads such as AI/ML, Big Data, and HPC jobs. It was the pr… | 10 | 1091 | abandoned |
| jorgecarleitao/arrow2 A Rust implementation of the Apache Arrow columnar in-memory format, supporting IO with Parquet, Avro, CSV, IPC, and Flight, plus compute k… | 10 | 1086 | abandoned |
| databricks/spark-sklearn A deprecated Python library from Databricks that integrates scikit-learn with Apache Spark, primarily to distribute grid-search cross-valid… | 10 | 1072 | abandoned |
| sitewhere/sitewhere SiteWhere is an open-source, multi-tenant IoT application enablement platform built on Java/Spring microservices, offering device and asset… | 10 | 1046 | abandoned |
| TIBCOSoftware/snappydata SnappyData (TIBCO ComputeDB) is a distributed, memory-optimized analytics database that fuses an in-memory hybrid OLTP/OLAP store inside Ap… | 23 | 1033 | abandoned |
| klbostee/dumbo Dumbo is a Python module that makes writing and running Hadoop Streaming programs easy, providing a convenient Python API for MapReduce pro… | 32 | 1031 | abandoned |
| facebookarchive/bistro Bistro is a C++ distributed task scheduler framework from Facebook that schedules and runs distributed tasks, including data-parallel jobs,… | 10 | 1027 | abandoned |
| SeaweedFS SeaweedFS is a fast distributed storage system written in Go that provides S3-compatible object storage, a POSIX-compliant distributed file… | 95 | 35551 | active |
| ThingsBoard ThingsBoard is an open-source IoT platform for device management, telemetry data collection, processing, and visualization. It supports sta… | 98 | 22494 | stable |
| questdb/questdb QuestDB is an open-source, high-performance time-series database with a column-oriented, SIMD-vectorized SQL engine written in zero-GC Java… | 97 | 17440 | stable |
| Data-Centric-AI-Community/fg-data-profiling A Python library that generates one-line data quality profiling and exploratory data analysis reports for Pandas and Spark DataFrames. It p… | 91 | 13719 | active |
| elastic/beats Elastic Beats is a family of lightweight data shippers written in Go that collect logs, metrics, network packets, audit data, and uptime si… | 95 | 12653 | active |
| ibis-project/ibis Ibis is a portable Python dataframe library that provides a single lazy dataframe API across more than 20 execution backends including Duck… | 87 | 6676 | active |
| Apache IoTDB Apache IoTDB is an open-source time-series database management system designed for industrial IoT, providing integrated collection, storage… | 94 | 6420 | stable |
| volcano-sh/volcano Volcano is a CNCF-hosted, Kubernetes-native batch scheduling system that extends kube-scheduler for high-performance workloads like AI/ML t… | 98 | 5993 | stable |
| AidLux (AidLearning) AidLux (originally AidLearning) is an AIoT development platform that runs a native Ubuntu Linux environment with GUI, deep learning tooling… | 63 | 5812 | active |
| googleapis/google-cloud-python The official monorepo of idiomatic Python client libraries for Google Cloud Platform services, covering APIs like BigQuery, Storage, AI Pla… | 95 | 5411 | stable |
| Memgraph Memgraph is a high-performance, in-memory graph database written in C++ that is Cypher-compatible and ACID-compliant, with built-in vector,… | 96 | 4579 | stable |
| Spark NLP Spark NLP is an open-source natural language processing library built natively on Apache Spark, providing scalable NLP annotations and tran… | 98 | 4160 | stable |
| aws/aws-sdk-pandas AWS SDK for pandas (awswrangler) is a Python library that extends pandas with high-level APIs for reading and writing data across AWS servi… | 91 | 4121 | active |
| borisdj/EFCore.BulkExtensions A .NET library extending Entity Framework Core with high-performance bulk CRUD operations (Insert, Update, Delete, Read, Upsert, Sync, Save… | 67 | 4000 | active |
| Unity Catalog Unity Catalog is an open-source, multimodal data and AI catalog that provides a unified metadata and governance layer for tables, files, fu… | 88 | 3555 | active |
| googleapis/google-cloud-node The official Google Cloud Client Library for Node.js, a monorepo containing idiomatic npm client libraries for interacting with individual … | 96 | 3209 | stable |
| polardb/PolarDB-for-PostgreSQL PolarDB for PostgreSQL is a cloud-native, PostgreSQL-compatible database system developed by Alibaba Cloud. It uses a shared-storage archit… | 91 | 3205 | active |
| lindb/lindb LinDB is an open-source distributed time series database written in Go, offering high performance, high availability, and horizontal scalab… | 62 | 3065 | active |
| hosseinmoein/DataFrame A high-performance, templatized C++ library providing a heterogeneous DataFrame container for in-memory data exploration, transformation, a… | 95 | 2990 | active |
| chDB chDB is an in-process SQL OLAP engine powered by ClickHouse, embeddable in Python (plus Go, Rust, Node.js, and Bun bindings) without runnin… | 97 | 2916 | active |
| Curve Curve is a cloud-native distributed storage system developed by NetEase and hosted as a CNCF sandbox project, supporting both block storage… | 23 | 2390 | active |
| openmeterio/openmeter OpenMeter is an open-source metering and billing platform for AI, API, and DevOps products that ingests high-volume usage events, aggregate… | 94 | 2341 | active |
| apconw/Aix-DB Aix-DB is an AI-powered data analysis system (ChatBI) built on LangChain/LangGraph with an MCP Skills multi-agent architecture, converting … | 74 | 2254 | active |
| jackchen0120/vueDataV A Vue + Echarts based data visualization platform offering cool big-screen dashboard templates and a reusable chart component library. It p… | 32 | 2178 | active |
| EvgSkv/logica Logica is an open-source declarative logic programming language in the Datalog family that compiles to SQL. It lets users write logic-style… | 77 | 2147 | active |
| apache/tinkerpop Apache TinkerPop is a graph computing framework providing the Gremlin traversal language, drivers, Gremlin Server, and a reference in-memor… | 77 | 2146 | stable |
| SREWorks SREWorks is Alibaba's open-source cloud-native operations and maintenance (O&M) SaaS platform built on Kubernetes, applying DataOps and AIO… | 44 | 2006 | active |
| MooseFS MooseFS is an open-source, POSIX-compliant, petabyte-scale network distributed file system written in C. It spreads data across commodity s… | 76 | 2003 | stable |
| AIStore AIStore (AIS) is a lightweight, distributed object storage stack built by NVIDIA specifically for AI workloads. It provides linear scalabil… | 95 | 1946 | active |
| Bruin Bruin is an end-to-end data platform whose CLI combines data ingestion (via its ingestr tool), SQL/Python/R transformations, and data quali… | 91 | 1774 | active |
| cnosdb/cnosdb CnosDB is an open-source, cloud-native distributed time-series database written in Rust, supporting standard SQL, schemaless writes, and ou… | 47 | 1759 | active |
| PolarDB-X PolarDB-X is a cloud native distributed SQL database built in Java with a shared-nothing architecture that decouples compute from storage. … | 85 | 1686 | stable |
| filodb/FiloDB FiloDB is a distributed, in-memory, Prometheus-compatible time series database written in Scala, designed for real-time ingestion of metric… | 67 | 1468 | active |
| sql-machine-learning/sqlflow SQLFlow is a compiler that extends SQL with AI-oriented syntax (training, prediction, evaluation, explanation, and mathematical programming… | 23 | 5190 | maintenance |
| apache/cassandra-python-driver The official Python client driver for Apache Cassandra (and DataStax Enterprise/Astra), communicating via Cassandra's binary protocol and C… | 70 | 1431 | stable |
| datavane/tis TIS is an AI-native data integration platform built on DataX, Flink, and Flink-CDC that provides a visual Web-UI for zero-code batch and re… | 88 | 1424 | active |
| Sensors Data SDK Sensors Analytics' official Android SDK for lightweight user behavior data collection, supporting manual code tracking, auto (full) trackin… | 89 | 1375 | stable |
| SpiderClub/weibospider A distributed web crawler for Sina Weibo (Chinese microblogging platform) built with Python, Celery, and requests. It scrapes user profiles… | 32 | 4790 | maintenance |
| Hopsworks Hopsworks is an open-source, data-intensive AI platform (an 'AI Lakehouse') built around a Python-centric Feature Store with online/offline… | 23 | 1310 | active |
| pmacct/pmacct pmacct is a set of multi-purpose passive network monitoring daemons written in C that collect, aggregate, replicate and export IP traffic a… | 66 | 1244 | active |
| OpenGemini openGemini is a cloud-native distributed time-series database written in Go, designed for storing and analyzing massive telemetry data such… | 78 | 1175 | active |
| sagframe/sagacity-sqltoy sqltoy-orm is a Java ORM framework that combines JPA-style object CRUD with powerful, flexible SQL query capabilities. It provides advanced… | 97 | 1150 | active |
| Tencent Kona JDK Tencent Kona 8 is a no-cost, production-ready distribution of OpenJDK 8 with long-term support and quarterly updates, serving as the defaul… | 86 | 1005 | stable |
| RaftLib/RaftLib RaftLib is an open-source C++ library for building parallel and concurrent data processing pipelines using a stream/dataflow programming mo… | 23 | 999 | active |
| dbt dbt Core is an open-source command-line tool that lets data analysts and engineers transform data in warehouses using SQL select statements… | 95 | 14039 | experimental |
| BlazingDB/blazingsql BlazingSQL is a GPU-accelerated distributed SQL engine for Python, built on the RAPIDS ecosystem and cuDF GPU DataFrames. It lets users run… | 23 | 2014 | maintenance |
| shzlw/poli Poli is a self-hosted business intelligence server for writing SQL queries and building interactive dashboards and reports. It connects to … | 23 | 1973 | maintenance |
| littleark/githut GitHut is a web-based visualization of programming language popularity across GitHub repositories, built on data from the GitHub Archive. I… | 32 | 1286 | maintenance |
| PixieDust PixieDust is an open-source Python helper library that acts as an add-on to Jupyter notebooks, simplifying data visualization and Spark pac… | 23 | 1040 | maintenance |
| madnight/githut A web application that visualizes GitHub language statistics, showing top programming languages, licenses, and pull request activity over t… | 32 | 1005 | maintenance |
| square/cube Cube is a system for collecting timestamped events and deriving metrics, built on MongoDB and Node.js, with time series visualization. It c… | 10 | 3871 | abandoned |
| airbnb/airpal Airpal is a web-based query execution UI for PrestoDB that lets users find tables, browse metadata, write queries, and export results as CS… | 10 | 2743 | abandoned |
| Cubes Cubes is a lightweight Python framework for Online Analytical Processing (OLAP), providing multidimensional analysis and browsing of aggreg… | 32 | 1479 | abandoned |
| citp/BlockSci BlockSci is a high-performance C++ tool and analysis library for blockchain science, featuring a custom in-memory blockchain database with … | 23 | 1396 | abandoned |
| twitter/fatcache Fatcache is a networked, memcached-protocol-compatible cache that uses SSD storage as an extension of DRAM, allowing terabyte-scale caching… | 10 | 1299 | abandoned |
← prev page 5 / 5