domain: big-data
477 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| microsoft/GraphEngine Microsoft Graph Engine is a distributed in-memory data processing engine built on a strongly-typed in-memory key-value store and a general-… | 32 | 2263 | maintenance |
| h2oai/h2o-2 H2O-2 is a Java-based distributed machine learning and math engine that scales statistics and predictive modeling over Big Data platforms l… | 10 | 2249 | maintenance |
| twitter-archive/distributedlog Apache DistributedLog is a high-throughput, low-latency replicated log service offering durability, replication, and strong ordering guaran… | 10 | 2203 | maintenance |
| dotnet/spark .NET for Apache Spark provides high-performance C# and F# bindings for Apache Spark, exposing DataFrames, SparkSQL, and Structured Streamin… | 77 | 2098 | maintenance |
| birdLark/LarkMidTable LarkMidTable (云雀) is a one-stop open-source data middleware platform covering metadata management, data warehouse development, data integra… | 32 | 2071 | maintenance |
| DTStack/flinkStreamSQL FlinkStreamSQL is a Java framework built on Apache Flink that extends Flink's real-time SQL with custom create table/view/function syntax a… | 23 | 2051 | maintenance |
| bytewax/bytewax Bytewax is a Python-first framework with a Rust-based distributed engine for stateful event and stream processing, inspired by Apache Flink… | 62 | 2047 | maintenance |
| Qihoo360/Quicksql Quicksql is a SQL analysis middleware that provides unified SQL querying across relational databases, non-relational databases, and SQL-les… | 23 | 2040 | maintenance |
| apache/drill Apache Drill is a distributed MPP (massively parallel processing) SQL query engine for self-describing data such as JSON, Parquet, and othe… | 67 | 2022 | maintenance |
| databricks/spark-deep-learning Deep Learning Pipelines for Apache Spark, now reduced to the HorovodRunner component for distributed deep learning training via Horovod on … | 23 | 1987 | maintenance |
| Qihoo360/poseidon Poseidon is a distributed log search platform from Qihoo 360 that builds inverted indexes over Hadoop/HDFS-stored logs and serves sub-secon… | 32 | 1981 | maintenance |
| HazyResearch/deepdive DeepDive is a Stanford-developed system for extracting structured data from unstructured sources and building knowledge bases using distant… | 23 | 1979 | maintenance |
| ICT-BDA/EasyML EasyML is a general-purpose dataflow-based machine learning platform where tasks are defined as directed acyclic graphs of operations. It i… | 23 | 1976 | maintenance |
| Tencent/plato Plato is Tencent's high-performance distributed graph computation framework, developed for graph processing and graph machine learning at W… | 23 | 1911 | maintenance |
| baidu/tera Tera is a high-performance distributed NoSQL database inspired by Google Bigtable, designed for real-time applications at internet scale. I… | 23 | 1902 | maintenance |
| uber/petastorm Petastorm is a Python data access library from Uber that enables single-machine or distributed training and evaluation of deep learning mod… | 58 | 1891 | maintenance |
| TurboWay/big_screen A simple Flask-based tool for building data visualization dashboards ('big screen' displays) in HTML. Users edit a Python data class with t… | 61 | 1877 | maintenance |
| h2oai/datatable datatable is a Python package for manipulating 2-dimensional tabular data structures (data frames), inspired by R's data.table. It emphasiz… | 65 | 1876 | maintenance |
| yougov/mongo-connector A Python-based pipeline tool that synchronizes data from a MongoDB cluster to target systems such as Solr, Elasticsearch, or another MongoD… | 23 | 1872 | maintenance |
| byzer-org/byzer-lang Byzer (formerly MLSQL) is a low-code, SQL-like distributed programming language and engine for data pipelines, analytics, and AI, built aro… | 23 | 1835 | maintenance |
| uber-common/jvm-profiler A Java Agent library from Uber that collects CPU, memory, IO metrics and stacktraces from JVM processes in a distributed way, with reporter… | 70 | 1804 | maintenance |
| RedisBloom/RedisBloom RedisBloom is a Redis module written in C that adds probabilistic data structures—Bloom filters, Cuckoo filters, Count-min sketch, Top-K, a… | 99 | 1784 | maintenance |
| embulk/embulk Embulk is an open-source, plugin-based parallel bulk data loader written in Java that transfers data between databases, storages, file form… | 62 | 1783 | maintenance |
| citusdata/cstore_fdw cstore_fdw is a PostgreSQL extension implementing a columnar store as a foreign data wrapper, inspired by the ORC format, designed for anal… | 23 | 1783 | maintenance |
| OryxProject/oryx Oryx 2 is a Java framework implementing the lambda architecture on Apache Spark and Apache Kafka, specialized for real-time large-scale mac… | 10 | 1782 | maintenance |
| xianrendzw/EasyReport EasyReport is a Java-based web reporting tool that converts SQL query results into HTML tables with row/column spanning, Excel export, char… | 23 | 1747 | maintenance |
| Qihoo360/hbox Hbox (formerly XLearning) is a scheduling platform that runs machine learning and deep learning frameworks like TensorFlow, MXNet, Caffe, a… | 47 | 1727 | maintenance |
| strapdata/elassandra Elassandra is a distribution of Apache Cassandra with Elasticsearch embedded in each node, providing full-text and spatial search over Cass… | 60 | 1714 | maintenance |
| dropbox/PyHive PyHive is a Python DB-API 2.0 and SQLAlchemy interface collection for Apache Hive, Presto, and Trino. It lets Python applications connect t… | 57 | 1693 | maintenance |
| Netflix/metacat Metacat is a unified metadata exploration API service from Netflix that federates metadata across data stores like Hive, RDS, Teradata, Red… | 99 | 1690 | maintenance |
| bytedance/bitsail BitSail is ByteDance's open-source distributed data integration engine built on Flink, supporting batch, streaming, and incremental data sy… | 10 | 1675 | maintenance |
| leo-project/leofs LeoFS is a highly available, distributed, eventually consistent object/blob storage system built in Erlang. It provides S3-compatible and R… | 50 | 1594 | maintenance |
| siddhi-io/siddhi Siddhi is a cloud-native stream processing and complex event processing (CEP) engine that executes Streaming SQL queries to capture events … | 72 | 1590 | maintenance |
| maxpumperla/elephas Elephas is a Python library that extends Keras to run distributed deep learning training on Apache Spark. It serializes Keras models on the… | 23 | 1577 | maintenance |
| re-data/re-data re_data is an open-source data reliability framework built as a dbt package for the modern data stack. It computes data quality metrics, de… | 23 | 1569 | maintenance |
| san089/goodreads_etl_pipeline An end-to-end ETL pipeline that ingests Goodreads API data into an AWS S3 data lake, transforms it with Spark on EMR, and loads it into a R… | 32 | 1542 | maintenance |
| pystorm/streamparse Streamparse is a Python library and CLI toolkit for running Python code in Apache Storm topologies, letting you write spouts and bolts with… | 58 | 1505 | maintenance |
| WallarooLabs/wally Wally (formerly Wallaroo) is a fast, distributed stream-processing framework written in Pony that handles distributed computing plumbing so… | 23 | 1482 | maintenance |
| apache/carbondata Apache CarbonData is an indexed columnar data file format and store for fast analytics on big data platforms like Apache Hadoop and Apache … | 75 | 1452 | maintenance |
| Tencent/TBase TBase is an enterprise-level distributed HTAP database based on the Postgres-XL project, providing highly consistent distributed transactio… | 46 | 1441 | maintenance |
| ucbrise/confluo Confluo is a C++ system for real-time monitoring and analysis of data streams, supporting high-throughput concurrent writes of millions of … | 32 | 1433 | maintenance |
| damklis/DataEngineeringProject An end-to-end data engineering project that scrapes news from RSS feeds via Airflow-scheduled Python scrapers and streams them through Kafk… | 32 | 1429 | maintenance |
| colinmarc/hdfs A native Go client for HDFS that connects directly to the namenode via its protocol buffers API, mimicking the stdlib os package interfaces… | 24 | 1403 | maintenance |
| distributedio/titan Titan is a distributed NoSQL database that provides a Redis protocol-compatible layer on top of TiKV, offering strong consistency, distribu… | 32 | 1402 | maintenance |
| loklak Loklak is a distributed Java server application that anonymously collects, indexes, and shares social media messages (especially tweets) us… | 23 | 1391 | maintenance |
| tensorflow/ecosystem A collection of templates and connectors for integrating TensorFlow with other open-source frameworks such as Kubernetes, Kubeflow, Spark, … | 10 | 1377 | maintenance |
| nathanmarz/cascalog Cascalog is a Clojure/Java data processing and querying library built on Hadoop, offering a high-level Datalog-like abstraction as a replac… | 32 | 1373 | maintenance |
| linkedin/dr-elephant Dr. Elephant is a self-serve performance monitoring and tuning tool for Apache Hadoop and Apache Spark jobs and workflows. It automatically… | 23 | 1369 | maintenance |
| scrapinghub/frontera Frontera is a Python web crawling framework that implements a scalable crawl frontier, storing and prioritizing links extracted by crawlers… | 34 | 1332 | maintenance |
| dongsuo/vue-data-board Vue-Data-Board is a DIY data analysis board built with Vue.js and ECharts, letting users create custom dashboards and data reports via drag… | 44 | 1327 | maintenance |
| didi/DDMQ DDMQ is a distributed messaging middleware built by DiDi on top of Apache RocketMQ, offering low-latency, high-throughput, highly available… | 10 | 1316 | maintenance |
| python-streamz/streamz Streamz is a Python library for building pipelines that manage continuous streams of real-time data. It supports complex pipelines with bra… | 66 | 1303 | maintenance |
| microsoft/Trill Trill is a high-performance, single-node, one-pass in-memory streaming analytics engine from Microsoft Research, built on a temporal data a… | 32 | 1271 | maintenance |
| AlgoTraders/stock-analysis-engine A distributed stock analysis and backtesting framework that ingests automated pricing data from IEX Cloud, Tradier, and FinViz and runs tho… | 32 | 1238 | maintenance |
| istresearch/scrapy-cluster Scrapy Cluster is a distributed web scraping framework built on Scrapy that uses Redis to coordinate crawl requests and Kafka as a data bus… | 10 | 1225 | maintenance |
| BriData/DBus DBus is a Java-based data bus platform that captures incremental changes from databases (via log-based CDC) and log sources in a non-intrus… | 23 | 1214 | maintenance |
| sryza/spark-timeseries A Scala/Java/Python library for time series analysis on Apache Spark, providing distributed abstractions for large time series datasets plu… | 32 | 1196 | maintenance |
| yahoo/egads EGADS (Extensible Generic Anomaly Detection System) is an open-source Java library for automatically detecting anomalies, outliers, and cha… | 32 | 1189 | maintenance |
| twosigma/flint Flint is Two Sigma's open-source time series library for Apache Spark, built around a time series aware TimeSeriesRDD data structure. It pr… | 32 | 1177 | maintenance |
| apache/griffin Apache Griffin is a model-driven data quality service platform for defining, executing, and reporting data quality measures across multiple… | 10 | 1172 | maintenance |
| lensacom/sparkit-learn Sparkit-learn provides scikit-learn's API and functionality on top of PySpark, operating on distributed RDDs of numpy arrays and sparse mat… | 23 | 1150 | maintenance |
| twitter/elephant-bird Twitter's Java library of LZO, Thrift, and Protocol Buffer-related Hadoop InputFormats, Pig LoadFuncs, Hive SerDes, and HBase utilities. It… | 23 | 1133 | maintenance |
| oeljeklaus-you/UserActionAnalyzePlatform A big data platform for e-commerce user behavior analysis built on Spark (Core, SQL, Streaming) with Java. It provides four analysis module… | 32 | 1127 | maintenance |
| ucarGroup/DataLink DataLink is a distributed, extensible data exchange platform for real-time incremental and offline full synchronization between heterogeneo… | 23 | 1120 | maintenance |
| travisjeffery/jocko Jocko is a distributed commit log service implementing the Kafka protocol in Go, with built-in coordination via Serf discovery and Raft con… | 60 | 5012 | experimental |
| Teradata/kylo Kylo is an open-source enterprise data lake management platform for self-service data ingest and preparation, with integrated metadata mana… | 32 | 1112 | maintenance |
| traildb/traildb TrailDB is a C library and tdb command-line tool for storing and querying series of discrete events (trails) in highly compressed, immutabl… | 32 | 1090 | maintenance |
| datumbox/datumbox-framework Datumbox is an open-source Machine Learning framework written in Java that enables rapid development of ML and statistical applications. It… | 32 | 1084 | maintenance |
| linkedin/goavro Goavro is a Go library that encodes and decodes Apache Avro data in both binary and textual JSON formats. It provides a stateless, goroutin… | 63 | 1071 | maintenance |
| wj596/go-mysql-transfer A standalone Go application that syncs MySQL data in real time by listening to the MySQL binlog as a fake replica. It transforms row change… | 23 | 1066 | maintenance |
| Blosc/c-blosc C-Blosc is a high-performance, lossless compression library for binary data that uses blocking, shuffling, SIMD, and multi-threading to com… | 48 | 1057 | maintenance |
| databricks/spark-csv A Scala library providing a CSV data source for Apache Spark 1.x, allowing CSV files to be read and queried as Spark SQL DataFrames with co… | 10 | 1057 | maintenance |
| twitter/cassovary Cassovary is a Scala-based big graph processing library for the JVM, designed to efficiently handle graphs with billions of nodes and edges… | 23 | 1052 | maintenance |
| facebookresearch/cc_net CCNet is a Python pipeline from Facebook AI Research for downloading, deduplicating, and cleaning Common Crawl web data into high-quality m… | 10 | 1045 | maintenance |
| mpplab/mnssp3 MNSSp3 is a privacy protection sharing platform for medical big data, built on the Malleable Network System Simulator (MNSS) platform. It a… | 32 | 1036 | maintenance |
| cloudera/livy Livy is an open-source REST service for interacting with Apache Spark remotely, supporting interactive Scala, Python, and R sessions as wel… | 32 | 1007 | maintenance |
| blaze/odo Odo is a Python library for migrating data between different containers, from in-memory structures like lists and pandas DataFrames to out-… | 23 | 1006 | maintenance |
| Cyb3rWard0g/HELK The Hunting ELK (HELK) is an open-source threat hunting platform built on the ELK stack (Elasticsearch, Logstash, Kibana) with advanced ana… | 23 | 3931 | experimental |
| rajasekarv/vega Vega (formerly native_spark) is a from-scratch reimplementation of Apache Spark's distributed data processing engine written in Rust. It ai… | 32 | 2229 | experimental |
| eBay/akutan Akutan is a distributed knowledge graph store (RDF/triple store) written in Go, designed to hold large graphs that don't fit on a single se… | 10 | 1651 | experimental |
| cswinter/LocustDB LocustDB is an experimental, embeddable analytics database written in Rust, designed for extremely fast query performance and storage effic… | 58 | 1648 | experimental |
| kkyon/botflow Botflow is a Python dataflow programming framework for building data pipelines using pipes and routes, with parallelism via coroutines and … | 62 | 1196 | experimental |
| apache/predictionio Apache PredictionIO is an open-source machine learning server built on Spark, HBase, Elasticsearch, and Hadoop that lets developers build, … | 10 | 12518 | abandoned |
| robinhood/faust Faust is a Python stream processing library that ports the ideas of Kafka Streams to Python, using asyncio for high-performance distributed… | 23 | 6825 | abandoned |
| thinkaurelius/titan Titan is a scalable distributed graph database optimized for storing and querying graphs with billions of vertices and edges, supporting th… | 32 | 5226 | abandoned |
| facebookarchive/scribe Scribe is a server for aggregating log data streamed in real time from a large number of clients, designed to be scalable and reliable. It … | 10 | 3908 | abandoned |
| alibaba/jstorm JStorm was Alibaba's enterprise stream processing engine, a Java-based distributed real-time computation framework compatible with Apache S… | 10 | 3871 | abandoned |
| apache/incubator-heron Apache Heron is a realtime, distributed, fault-tolerant stream processing engine originally developed at Twitter as a successor to Apache S… | 10 | 3628 | abandoned |
| mozilla-services/heka Heka is a Go-based tool for collecting data from many sources, performing in-flight processing, and delivering results to multiple destinat… | 10 | 3394 | abandoned |
| LiuXingMing/SinaSpider A Python web crawler for Sina Weibo (Chinese microblog) built on Scrapy, with three versions: a standalone spider, a distributed version us… | 32 | 3285 | abandoned |
| microsoft/DMTK DMTK is Microsoft's Distributed Machine Learning Toolkit, an umbrella project hosting a parameter server framework (Multiverso) plus distri… | 10 | 2738 | abandoned |
| intel/BigDL BigDL is Intel's distributed deep learning library that scales TensorFlow, Keras, and PyTorch workloads on Apache Spark, Flink, and Ray, wi… | 10 | 2698 | abandoned |
| voldemort/voldemort Voldemort is a distributed key-value storage system modeled on Amazon's Dynamo, providing automatic replication, partitioning, versioning, … | 23 | 2686 | abandoned |
| FeatureBaseDB/featurebase FeatureBase (formerly Pilosa) is a distributed analytical database built entirely on bitmap indexes, offering low-latency SQL queries over … | 10 | 2521 | abandoned |
| ByConity/ByConity ByConity is an open-source cloud-native distributed SQL data warehouse built on the ClickHouse 21.8 codebase, developed by ByteDance/Volcan… | 10 | 2239 | abandoned |
| mozilla/MozDef MozDef is Mozilla's Enterprise Defense Platform, an open-source SIEM-style application for automating security incident handling and enabli… | 10 | 2161 | abandoned |
| twitter/summingbird Summingbird is a Scala library for writing MapReduce aggregation programs that look like native collection transformations and run on distr… | 10 | 2123 | abandoned |
| onyx-platform/onyx Onyx is a masterless, fault-tolerant distributed computation system written in pure Clojure that supports both batch and stream processing … | 10 | 2051 | abandoned |
| Tencent/TubeMQ TubeMQ was Tencent's high-performance message queue, donated to the Apache Software Foundation in 2019 and renamed Apache InLong. This repo… | 23 | 1996 | abandoned |
| facebookarchive/LogDevice LogDevice is a scalable, fault-tolerant distributed log storage system written in C++ by Facebook, storing append-only record streams repli… | 10 | 1901 | abandoned |