Open-source big-data software
483 open-source libraries, tools and applications tagged big-data, ranked by how closely big-data is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
483 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| steveloughran/winutils Windows native binaries (winutils.exe and Hadoop DLLs) for various Hadoop versions, built from the same git commits as official ASF release… | 23 | 2634 | maintenance |
| Yelp/mrjob mrjob is a Python library for writing and running Hadoop Streaming MapReduce jobs, with support for Amazon EMR, Google Cloud Dataproc, self… | 62 | 2613 | maintenance |
| apache/logging-flume Apache Flume is a distributed, reliable, and available service for efficiently collecting, aggregating, and moving large amounts of log-lik… | 76 | 2570 | maintenance |
| justinzm/gopup GoPUP is a Python library that provides convenient interfaces to a wide range of public Chinese data sources, including Baidu/Weibo/Google … | 23 | 2545 | maintenance |
| geekyouth/SZT-bigdata A big data passenger flow analysis system for the Shenzhen Metro, built on Shenzhen Tong smart-card swipe data. It demonstrates ETL and ana… | 56 | 2475 | maintenance |
| cdarlint/winutils A repository of precompiled Windows binaries (winutils.exe, hadoop.dll, hdfs.dll) needed to run Hadoop natively on Windows. It continues th… | 32 | 2299 | maintenance |
| salesforce/TransmogrifAI TransmogrifAI is an AutoML library written in Scala that runs on Apache Spark for building modular, reusable, strongly typed machine learni… | 57 | 2276 | maintenance |
| microsoft/GraphEngine Microsoft Graph Engine is a distributed in-memory data processing engine built on a strongly-typed in-memory key-value store and a general-… | 32 | 2266 | maintenance |
| H2O H2O-2 is a Java-based distributed machine learning and math engine that scales statistics and predictive modeling over Big Data platforms l… | 10 | 2240 | maintenance |
| twitter-archive/distributedlog Apache DistributedLog is a high-throughput, low-latency replicated log service offering durability, replication, and strong ordering guaran… | 10 | 2202 | maintenance |
| dotnet/spark .NET for Apache Spark provides high-performance C# and F# bindings for Apache Spark, exposing DataFrames, SparkSQL, and Structured Streamin… | 74 | 2095 | maintenance |
| birdLark/LarkMidTable LarkMidTable (云雀) is a one-stop open-source data middleware platform covering metadata management, data warehouse development, data integra… | 32 | 2073 | maintenance |
| bytewax/bytewax Bytewax is a Python-first framework with a Rust-based distributed engine for stateful event and stream processing, inspired by Apache Flink… | 68 | 2056 | maintenance |
| DTStack/flinkStreamSQL FlinkStreamSQL is a Java framework built on Apache Flink that extends Flink's real-time SQL with custom create table/view/function syntax a… | 23 | 2045 | maintenance |
| Qihoo360/Quicksql Quicksql is a SQL analysis middleware that provides unified SQL querying across relational databases, non-relational databases, and SQL-les… | 23 | 2037 | maintenance |
| apache/drill Apache Drill is a distributed MPP (massively parallel processing) SQL query engine for self-describing data such as JSON, Parquet, and othe… | 67 | 2024 | maintenance |
| databricks/spark-deep-learning Deep Learning Pipelines for Apache Spark, now reduced to the HorovodRunner component for distributed deep learning training via Horovod on … | 23 | 1987 | maintenance |
| Qihoo360/poseidon Poseidon is a distributed log search platform from Qihoo 360 that builds inverted indexes over Hadoop/HDFS-stored logs and serves sub-secon… | 32 | 1980 | maintenance |
| HazyResearch/deepdive DeepDive is a Stanford-developed system for extracting structured data from unstructured sources and building knowledge bases using distant… | 23 | 1979 | maintenance |
| ICT-BDA/EasyML EasyML is a general-purpose dataflow-based machine learning platform where tasks are defined as directed acyclic graphs of operations. It i… | 23 | 1979 | maintenance |
| Tencent/plato Plato is Tencent's high-performance distributed graph computation framework, developed for graph processing and graph machine learning at W… | 23 | 1912 | maintenance |
| baidu/tera Tera is a high-performance distributed NoSQL database inspired by Google Bigtable, designed for real-time applications at internet scale. I… | 23 | 1903 | maintenance |
| uber/petastorm Petastorm is a Python data access library from Uber that enables single-machine or distributed training and evaluation of deep learning mod… | 52 | 1894 | maintenance |
| TurboWay/big_screen A simple Flask-based tool for building data visualization dashboards ('big screen' displays) in HTML. Users edit a Python data class with t… | 57 | 1883 | maintenance |
| h2oai/datatable datatable is a Python package for manipulating 2-dimensional tabular data structures (data frames), inspired by R's data.table. It emphasiz… | 64 | 1877 | maintenance |
| mongo-connector A Python-based pipeline tool that synchronizes data from a MongoDB cluster to target systems such as Solr, Elasticsearch, or another MongoD… | 23 | 1871 | maintenance |
| byzer-org/byzer-lang Byzer (formerly MLSQL) is a low-code, SQL-like distributed programming language and engine for data pipelines, analytics, and AI, built aro… | 23 | 1835 | maintenance |
| uber-common/jvm-profiler A Java Agent library from Uber that collects CPU, memory, IO metrics and stacktraces from JVM processes in a distributed way, with reporter… | 66 | 1801 | maintenance |
| Embulk Embulk is an open-source, plugin-based parallel bulk data loader written in Java that transfers data between databases, storages, file form… | 67 | 1784 | maintenance |
| RedisBloom RedisBloom is a Redis module written in C that adds probabilistic data structures—Bloom filters, Cuckoo filters, Count-min sketch, Top-K, a… | 95 | 1783 | maintenance |
| citusdata/cstore_fdw cstore_fdw is a PostgreSQL extension implementing a columnar store as a foreign data wrapper, inspired by the ORC format, designed for anal… | 23 | 1783 | maintenance |
| OryxProject/oryx Oryx 2 is a Java framework implementing the lambda architecture on Apache Spark and Apache Kafka, specialized for real-time large-scale mac… | 10 | 1782 | maintenance |
| xianrendzw/EasyReport EasyReport is a Java-based web reporting tool that converts SQL query results into HTML tables with row/column spanning, Excel export, char… | 23 | 1749 | maintenance |
| Qihoo360/hbox Hbox (formerly XLearning) is a scheduling platform that runs machine learning and deep learning frameworks like TensorFlow, MXNet, Caffe, a… | 43 | 1728 | maintenance |
| strapdata/elassandra Elassandra is a distribution of Apache Cassandra with Elasticsearch embedded in each node, providing full-text and spatial search over Cass… | 56 | 1714 | maintenance |
| dropbox/PyHive PyHive is a Python DB-API 2.0 and SQLAlchemy interface collection for Apache Hive, Presto, and Trino. It lets Python applications connect t… | 54 | 1694 | maintenance |
| Netflix/metacat Metacat is a unified metadata exploration API service from Netflix that federates metadata across data stores like Hive, RDS, Teradata, Red… | 96 | 1692 | maintenance |
| bytedance/bitsail BitSail is ByteDance's open-source distributed data integration engine built on Flink, supporting batch, streaming, and incremental data sy… | 10 | 1674 | maintenance |
| leo-project/leofs LeoFS is a highly available, distributed, eventually consistent object/blob storage system built in Erlang. It provides S3-compatible and R… | 46 | 1595 | maintenance |
| siddhi-io/siddhi Siddhi is a cloud-native stream processing and complex event processing (CEP) engine that executes Streaming SQL queries to capture events … | 66 | 1593 | maintenance |
| maxpumperla/elephas Elephas is a Python library that extends Keras to run distributed deep learning training on Apache Spark. It serializes Keras models on the… | 23 | 1574 | maintenance |
| re_data re_data is an open-source data reliability framework built as a dbt package for the modern data stack. It computes data quality metrics, de… | 23 | 1572 | maintenance |
| san089/goodreads_etl_pipeline An end-to-end ETL pipeline that ingests Goodreads API data into an AWS S3 data lake, transforms it with Spark on EMR, and loads it into a R… | 32 | 1545 | maintenance |
| streamparse Streamparse is a Python library and CLI toolkit for running Python code in Apache Storm topologies, letting you write spouts and bolts with… | 54 | 1503 | maintenance |
| WallarooLabs/wally Wally (formerly Wallaroo) is a fast, distributed stream-processing framework written in Pony that handles distributed computing plumbing so… | 65 | 1483 | maintenance |
| apache/carbondata Apache CarbonData is an indexed columnar data file format and store for fast analytics on big data platforms like Apache Hadoop and Apache … | 76 | 1451 | maintenance |
| Tencent/TBase TBase is an enterprise-level distributed HTAP database based on the Postgres-XL project, providing highly consistent distributed transactio… | 42 | 1442 | maintenance |
| damklis/DataEngineeringProject An end-to-end data engineering project that scrapes news from RSS feeds via Airflow-scheduled Python scrapers and streams them through Kafk… | 32 | 1440 | maintenance |
| ucbrise/confluo Confluo is a C++ system for real-time monitoring and analysis of data streams, supporting high-throughput concurrent writes of millions of … | 74 | 1431 | maintenance |
| distributedio/titan Titan is a distributed NoSQL database that provides a Redis protocol-compatible layer on top of TiKV, offering strong consistency, distribu… | 32 | 1403 | maintenance |
| colinmarc/hdfs A native Go client for HDFS that connects directly to the namenode via its protocol buffers API, mimicking the stdlib os package interfaces… | 23 | 1401 | maintenance |
| loklak Loklak is a distributed Java server application that anonymously collects, indexes, and shares social media messages (especially tweets) us… | 23 | 1393 | maintenance |
| tensorflow/ecosystem A collection of templates and connectors for integrating TensorFlow with other open-source frameworks such as Kubernetes, Kubeflow, Spark, … | 10 | 1377 | maintenance |
| nathanmarz/cascalog Cascalog is a Clojure/Java data processing and querying library built on Hadoop, offering a high-level Datalog-like abstraction as a replac… | 32 | 1373 | maintenance |
| linkedin/dr-elephant Dr. Elephant is a self-serve performance monitoring and tuning tool for Apache Hadoop and Apache Spark jobs and workflows. It automatically… | 23 | 1370 | maintenance |
| scrapinghub/frontera Frontera is a Python web crawling framework that implements a scalable crawl frontier, storing and prioritizing links extracted by crawlers… | 30 | 1334 | maintenance |
| dongsuo/vue-data-board Vue-Data-Board is a DIY data analysis board built with Vue.js and ECharts, letting users create custom dashboards and data reports via drag… | 40 | 1327 | maintenance |
| didi/DDMQ DDMQ is a distributed messaging middleware built by DiDi on top of Apache RocketMQ, offering low-latency, high-throughput, highly available… | 10 | 1314 | maintenance |
| python-streamz/streamz Streamz is a Python library for building pipelines that manage continuous streams of real-time data. It supports complex pipelines with bra… | 75 | 1303 | maintenance |
| Trill Trill is a high-performance, single-node, one-pass in-memory streaming analytics engine from Microsoft Research, built on a temporal data a… | 32 | 1271 | maintenance |
| AlgoTraders/stock-analysis-engine A distributed stock analysis and backtesting framework that ingests automated pricing data from IEX Cloud, Tradier, and FinViz and runs tho… | 32 | 1247 | maintenance |
| istresearch/scrapy-cluster Scrapy Cluster is a distributed web scraping framework built on Scrapy that uses Redis to coordinate crawl requests and Kafka as a data bus… | 10 | 1222 | maintenance |
| BriData/DBus DBus is a Java-based data bus platform that captures incremental changes from databases (via log-based CDC) and log sources in a non-intrus… | 23 | 1212 | maintenance |
| spark-timeseries A Scala/Java/Python library for time series analysis on Apache Spark, providing distributed abstractions for large time series datasets plu… | 32 | 1194 | maintenance |
| yahoo/egads EGADS (Extensible Generic Anomaly Detection System) is an open-source Java library for automatically detecting anomalies, outliers, and cha… | 32 | 1190 | maintenance |
| twosigma/flint Flint is Two Sigma's open-source time series library for Apache Spark, built around a time series aware TimeSeriesRDD data structure. It pr… | 32 | 1181 | maintenance |
| apache/griffin Apache Griffin is a model-driven data quality service platform for defining, executing, and reporting data quality measures across multiple… | 10 | 1171 | maintenance |
| lensacom/sparkit-learn Sparkit-learn provides scikit-learn's API and functionality on top of PySpark, operating on distributed RDDs of numpy arrays and sparse mat… | 23 | 1150 | maintenance |
| twitter/elephant-bird Twitter's Java library of LZO, Thrift, and Protocol Buffer-related Hadoop InputFormats, Pig LoadFuncs, Hive SerDes, and HBase utilities. It… | 23 | 1133 | maintenance |
| oeljeklaus-you/UserActionAnalyzePlatform A big data platform for e-commerce user behavior analysis built on Spark (Core, SQL, Streaming) with Java. It provides four analysis module… | 32 | 1132 | maintenance |
| ucarGroup/DataLink DataLink is a distributed, extensible data exchange platform for real-time incremental and offline full synchronization between heterogeneo… | 23 | 1118 | maintenance |
| Teradata/kylo Kylo is an open-source enterprise data lake management platform for self-service data ingest and preparation, with integrated metadata mana… | 32 | 1116 | maintenance |
| travisjeffery/jocko Jocko is a distributed commit log service implementing the Kafka protocol in Go, with built-in coordination via Serf discovery and Raft con… | 57 | 5009 | experimental |
| traildb/traildb TrailDB is a C library and tdb command-line tool for storing and querying series of discrete events (trails) in highly compressed, immutabl… | 32 | 1090 | maintenance |
| datumbox/datumbox-framework Datumbox is an open-source Machine Learning framework written in Java that enables rapid development of ML and statistical applications. It… | 32 | 1082 | maintenance |
| linkedin/goavro Goavro is a Go library that encodes and decodes Apache Avro data in both binary and textual JSON formats. It provides a stateless, goroutin… | 57 | 1073 | maintenance |
| wj596/go-mysql-transfer A standalone Go application that syncs MySQL data in real time by listening to the MySQL binlog as a fake replica. It transforms row change… | 23 | 1066 | maintenance |
| C-Blosc2 C-Blosc is a high-performance, lossless compression library for binary data that uses blocking, shuffling, SIMD, and multi-threading to com… | 64 | 1059 | maintenance |
| databricks/spark-csv A Scala library providing a CSV data source for Apache Spark 1.x, allowing CSV files to be read and queried as Spark SQL DataFrames with co… | 10 | 1056 | maintenance |
| twitter/cassovary Cassovary is a Scala-based big graph processing library for the JVM, designed to efficiently handle graphs with billions of nodes and edges… | 23 | 1053 | maintenance |
| facebookresearch/cc_net CCNet is a Python pipeline from Facebook AI Research for downloading, deduplicating, and cleaning Common Crawl web data into high-quality m… | 10 | 1045 | maintenance |
| mpplab/mnssp3 MNSSp3 is a privacy protection sharing platform for medical big data, built on the Malleable Network System Simulator (MNSS) platform. It a… | 32 | 1036 | maintenance |
| cloudera/livy Livy is an open-source REST service for interacting with Apache Spark remotely, supporting interactive Scala, Python, and R sessions as wel… | 32 | 1006 | maintenance |
| blaze/odo Odo is a Python library for migrating data between different containers, from in-memory structures like lists and pandas DataFrames to out-… | 23 | 1005 | maintenance |
| Cyb3rWard0g/HELK The Hunting ELK (HELK) is an open-source threat hunting platform built on the ELK stack (Elasticsearch, Logstash, Kibana) with advanced ana… | 23 | 3931 | experimental |
| rajasekarv/vega Vega (formerly native_spark) is a from-scratch reimplementation of Apache Spark's distributed data processing engine written in Rust. It ai… | 32 | 2229 | experimental |
| eBay/akutan Akutan is a distributed knowledge graph store (RDF/triple store) written in Go, designed to hold large graphs that don't fit on a single se… | 10 | 1653 | experimental |
| cswinter/LocustDB LocustDB is an experimental, embeddable analytics database written in Rust, designed for extremely fast query performance and storage effic… | 54 | 1648 | experimental |
| kkyon/botflow Botflow is a Python dataflow programming framework for building data pipelines using pipes and routes, with parallelism via coroutines and … | 58 | 1193 | experimental |
| Apache PredictionIO Apache PredictionIO is an open-source machine learning server built on Spark, HBase, Elasticsearch, and Hadoop that lets developers build, … | 10 | 12520 | abandoned |
| robinhood/faust Faust is a Python stream processing library that ports the ideas of Kafka Streams to Python, using asyncio for high-performance distributed… | 23 | 6823 | abandoned |
| thinkaurelius/titan Titan is a scalable distributed graph database optimized for storing and querying graphs with billions of vertices and edges, supporting th… | 32 | 5229 | abandoned |
| facebookarchive/scribe Scribe is a server for aggregating log data streamed in real time from a large number of clients, designed to be scalable and reliable. It … | 10 | 3906 | abandoned |
| alibaba/jstorm JStorm was Alibaba's enterprise stream processing engine, a Java-based distributed real-time computation framework compatible with Apache S… | 10 | 3867 | abandoned |
| apache/incubator-heron Apache Heron is a realtime, distributed, fault-tolerant stream processing engine originally developed at Twitter as a successor to Apache S… | 10 | 3624 | abandoned |
| mozilla-services/heka Heka is a Go-based tool for collecting data from many sources, performing in-flight processing, and delivering results to multiple destinat… | 10 | 3394 | abandoned |
| LiuXingMing/SinaSpider A Python web crawler for Sina Weibo (Chinese microblog) built on Scrapy, with three versions: a standalone spider, a distributed version us… | 32 | 3283 | abandoned |
| DMTK DMTK is Microsoft's Distributed Machine Learning Toolkit, an umbrella project hosting a parameter server framework (Multiverso) plus distri… | 10 | 2735 | abandoned |
| intel/BigDL BigDL is Intel's distributed deep learning library that scales TensorFlow, Keras, and PyTorch workloads on Apache Spark, Flink, and Ray, wi… | 10 | 2699 | abandoned |
| voldemort/voldemort Voldemort is a distributed key-value storage system modeled on Amazon's Dynamo, providing automatic replication, partitioning, versioning, … | 23 | 2686 | abandoned |