Open-source etl software
673 open-source libraries, tools and applications tagged etl, ranked by how closely etl is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
673 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| OpenBB OpenBB is an open-source Python library that provides a unified data integration layer for financial, economic, and crypto data. It powers … | 85 | 73550 | active |
| Pathway Live Data Framework Pathway is a Python ETL framework designed for stream processing, real-time analytics, and building LLM pipelines. It provides a high‑level… | 94 | 62243 | active |
| pandas pandas is a Python library that provides fast, flexible, and expressive data structures for labeled data. It is widely used for data cleani… | 94 | 49994 | active |
| pandas pandas is a fast, flexible Python library providing labeled data structures like the DataFrame for practical, real-world data analysis and … | 94 | 49994 | stable |
| Qlib Qlib is an AI‑oriented quantitative investment platform written in Python. It provides end‑to‑end tools for data preparation, model trainin… | 66 | 48694 | active |
| Apache Airflow Apache Airflow is an open-source platform for programmatically authoring, scheduling, and monitoring workflows as directed acyclic graphs (… | 98 | 47148 | stable |
| Apache Spark Apache Spark is a unified analytics engine for large-scale data processing that provides high‑level APIs in Scala, Java, Python, and R. It … | 77 | 44159 | active |
| Apache Spark Apache Spark is a unified analytics engine for large-scale data processing, providing high-level APIs in Scala, Java, Python, and R over an… | 77 | 44159 | stable |
| Polars Polars is a high-performance DataFrame library written in Rust with bindings for Python, Node.js, R and more. It provides fast, lazy and ea… | 92 | 39857 | active |
| Polars Polars is an extremely fast analytical query engine for DataFrames written in Rust, with multi-threaded vectorized execution, lazy query op… | 92 | 39857 | stable |
| Apache Kafka Apache Kafka is an open-source distributed event streaming platform for building high-performance data pipelines, streaming analytics, and … | 76 | 33918 | stable |
| Kestra Kestra is an open-source, event-driven orchestration and scheduling platform for data, AI, and infrastructure workflows, written in Java. I… | 95 | 29837 | stable |
| alibaba/canal Canal is an Alibaba open-source component that parses MySQL binlog to provide incremental data subscription and consumption. It masquerades… | 62 | 29740 | stable |
| Apache Flink Apache Flink is an open-source distributed stream processing framework for stateful computations over unbounded and bounded data streams, w… | 77 | 26413 | stable |
| dataease/dataease DataEase is an open‑source business intelligence platform that lets users connect to many data sources and create interactive charts and da… | 98 | 24599 | active |
| Vector Vector is a high‑performance, end‑to‑end observability data pipeline written in Rust. It collects, transforms, and routes logs, metrics and… | 93 | 22657 | active |
| Vector Vector is a high-performance observability data pipeline written in Rust that collects, transforms, and routes logs, metrics, and traces to… | 93 | 22657 | active |
| Airbyte Airbyte is an open-source data movement platform providing 600+ connectors for replicating data from APIs, databases, and files into wareho… | 77 | 22234 | active |
| huggingface/datasets 🤗 Datasets is a Python library that provides one‑line loading of thousands of public datasets and powerful tools for preprocessing and stre… | 99 | 22061 | active |
| spotify/luigi Luigi is a Python package for building complex pipelines of long-running batch jobs. It handles dependency resolution, workflow management,… | 84 | 18780 | stable |
| influxdata/telegraf Telegraf is an open-source agent for collecting, processing, aggregating, and writing metrics, logs, and other telemetry data. It ships as … | 100 | 17865 | stable |
| alibaba/DataX DataX is Alibaba's open-source offline data synchronization framework, the open version of Alibaba Cloud DataWorks data integration. It syn… | 60 | 17364 | stable |
| apache/arrow Apache Arrow is a language-independent columnar in-memory data format specification plus multi-language libraries (C++, Python/PyArrow, Jav… | 95 | 17212 | stable |
| Presto Presto is a distributed SQL query engine for running interactive queries against large datasets across heterogeneous data sources such as H… | 97 | 16757 | stable |
| dagster-io/dagster Dagster is an open-source Python data orchestration platform for developing, producing, and observing data assets, with integrated lineage,… | 95 | 16287 | active |
| apache/hadoop Apache Hadoop is an open-source framework for reliable, scalable distributed storage (HDFS) and processing (MapReduce, YARN) of large data … | 77 | 15694 | stable |
| Unstructured Unstructured is an open-source ETL library and platform for converting complex documents (PDF, DOCX, HTML, images, and 65+ file types) into… | 95 | 15556 | active |
| ConardLi/easy-dataset Easy Dataset is a self-hosted web application for building high-quality structured datasets for LLM fine-tuning, RAG, and model evaluation.… | 66 | 14989 | active |
| elastic/logstash Logstash is an open-source server-side data processing pipeline that ingests data from multiple sources simultaneously, transforms it, and … | 95 | 14966 | stable |
| Apache DolphinScheduler Apache DolphinScheduler is a modern data orchestration platform for building high-performance workflows with low-code drag-and-drop tooling… | 92 | 14510 | stable |
| Eclipse Deeplearning4j Eclipse Deeplearning4J is an open-source deep learning framework and ecosystem for the JVM, including the ND4J linear algebra library, the … | 76 | 14269 | active |
| Dask Dask is a flexible parallel and distributed computing library for Python that scales pandas, NumPy, scikit-learn, and other PyData tools to… | 96 | 13934 | stable |
| Fluentd Fluentd is an open-source data collector that unifies log and event collection from many sources and routes them to files, databases, cloud… | 94 | 13606 | stable |
| Debezium Debezium is an open source distributed platform for change data capture (CDC) that streams row-level database changes as events, most commo… | 77 | 13237 | stable |
| datahub-project/datahub DataHub is an open-source data catalog and metadata platform providing data discovery, governance, lineage, and observability across an org… | 99 | 12852 | active |
| Laravel Excel Laravel Excel (maatwebsite/excel) is a Laravel package providing an elegant wrapper around PhpSpreadsheet for exporting and importing Excel… | 96 | 12704 | stable |
| elastic/beats Elastic Beats is a family of lightweight data shippers written in Go that collect logs, metrics, network packets, audit data, and uptime si… | 95 | 12653 | active |
| OpenRefine OpenRefine is a free, open-source desktop application (running a local web server accessed via browser) for cleaning, transforming, and enr… | 87 | 12016 | stable |
| fivetran/great_expectations Great Expectations (GX Core) is a Python library for validating, documenting, and profiling data using declarative, human-readable assertio… | 95 | 11871 | active |
| QUANTAXIS QUANTAXIS is a Python quantitative finance framework providing local data fetching, backtesting, simulated and live trading, visualization,… | 66 | 11252 | active |
| Kedro Kedro is an open-source Python framework for building production-ready data engineering and data science pipelines using software engineeri… | 95 | 11028 | stable |
| Lago Lago is an open-source metering and usage-based billing platform that ingests usage events, manages subscriptions and pricing, and generate… | 95 | 10699 | active |
| modin-project/modin Modin is a drop-in replacement for pandas that scales DataFrame operations across all CPU cores using Ray, Dask, or Unidist as execution en… | 61 | 10389 | active |
| TPOT TPOT (Tree-based Pipeline Optimization Tool) is a Python automated machine learning library that optimizes scikit-learn machine learning pi… | 40 | 10053 | active |
| johnkerl/miller Miller (mlr) is a command-line tool like awk, sed, cut, join, and sort but designed for name-indexed data such as CSV, TSV, JSON, JSON Line… | 94 | 10044 | stable |
| NVIDIA/cudf cuDF is a GPU-accelerated DataFrame library for tabular data processing, part of NVIDIA's RAPIDS suite. It provides a pandas-compatible Pyt… | 95 | 9780 | stable |
| dataabc/weiboSpider A Python command-line crawler that scrapes posts and profile data from Sina Weibo users, writing results to txt/csv/json files or MySQL/Mon… | 58 | 9733 | active |
| Apache SeaTunnel Apache SeaTunnel is a distributed, high-performance data integration platform for synchronizing massive amounts of data across hundreds of … | 90 | 9700 | stable |
| blue-yonder/tsfresh tsfresh is a Python package that automatically extracts hundreds of features from time series using algorithms from statistics, signal proc… | 74 | 9470 | stable |
| RisingWave RisingWave is an event streaming platform that continuously ingests data from databases, event streams, and webhooks, processes it incremen… | 100 | 9381 | active |
| ML.NET ML.NET is an open-source, cross-platform machine learning framework for .NET that lets developers build, train, and deploy custom ML models… | 78 | 9356 | active |
| Delta Lake Delta Lake is an open-source storage framework and table format that brings ACID transactions, scalable metadata handling, schema enforceme… | 95 | 9053 | stable |
| mage-ai/mage-ai Mage OSS is a self-hosted data pipeline tool for building, running, and orchestrating ETL/ELT workflows in a notebook-style UI using Python… | 91 | 8835 | active |
| redpanda-data/connect Redpanda Connect (formerly Benthos) is a declarative stream processor that moves data between hundreds of sources and sinks with transforma… | 94 | 8778 | active |
| apache/beam Apache Beam is an open-source unified programming model and SDK set (Java, Python, Go, SQL, TypeScript) for defining batch and streaming da… | 95 | 8688 | stable |
| pentaho/pentaho-kettle Pentaho Data Integration (Kettle) is an open-source ETL application for designing and running data extraction, transformation, and loading … | 66 | 8400 | active |
| OpenDCAI/DataFlow DataFlow is a Python framework for preparing, cleaning, and synthesizing data for LLM training and RAG using composable LLM-based operators… | 77 | 8201 | active |
| turbot/steampipe Steampipe is a zero-ETL CLI tool that exposes APIs and cloud services as relational database tables so they can be queried live with standa… | 97 | 7968 | active |
| elasticsearch-dump/elasticsearch-dump A Node.js CLI tool (elasticdump) for importing and exporting Elasticsearch and OpenSearch indices, moving data between clusters or to/from … | 91 | 7941 | active |
| alteryx/featuretools Featuretools is an open-source Python library for automated feature engineering, using Deep Feature Synthesis (DFS) to generate features fr… | 65 | 7687 | active |
| microsoft/graphrag GraphRAG is a Python library and CLI pipeline from Microsoft Research that extracts a knowledge graph from unstructured text, builds commun… | 92 | 36292 | maintenance |
| Enso Analytics Enso Analytics is a self-service data preparation and analysis platform for data teams, built on a hybrid visual/textual functional program… | 84 | 7442 | active |
| Feast Feast is an open-source feature store for machine learning that manages offline stores for historical training data and low-latency online … | 99 | 7327 | stable |
| AlaSQL/alasql AlaSQL is a lightweight in-memory SQL database written in pure JavaScript that runs in the browser, Node.js, and mobile apps. It queries bo… | 100 | 7291 | active |
| datajuicer/data-juicer Data-Juicer is a Python library and data processing system for cleaning, deduplicating, synthesizing, and analyzing data for foundation mod… | 94 | 7132 | active |
| Snowplow Snowplow is a Customer Data Infrastructure platform that collects, validates, enriches, and streams event-level behavioral data from web, m… | 59 | 7034 | stable |
| ibis-project/ibis Ibis is a portable Python dataframe library that provides a single lazy dataframe API across more than 20 execution backends including Duck… | 87 | 6676 | active |
| apache/zeppelin Apache Zeppelin is a web-based notebook for interactive, data-driven analytics and collaborative documents. It supports SQL, Scala, Python,… | 76 | 6663 | stable |
| airweave-ai/airweave Airweave is an open-source context retrieval layer that connects to apps, databases, and documents and exposes their data through a single … | 10 | 6558 | active |
| dimitri/pgloader pgloader is a data loading and migration tool for PostgreSQL built on the COPY command. It migrates entire databases from MySQL, SQLite, MS… | 80 | 6544 | active |
| cloudquery/cloudquery CloudQuery is an open-source CLI and plugin-based data pipeline tool that extracts cloud infrastructure configuration and security data fro… | 95 | 6542 | active |
| nteract/papermill Papermill is a Python library and CLI tool for parameterizing, executing, and analyzing Jupyter notebooks. It enables running notebooks pro… | 70 | 6486 | active |
| apache/flink-cdc Flink CDC is a distributed streaming data integration tool built on Apache Flink that captures change data from databases like MySQL and Po… | 79 | 6482 | active |
| csvkit csvkit is a suite of Python-based command-line tools for converting to and working with CSV files. It includes utilities like csvcut, csvst… | 75 | 6415 | stable |
| Materialize Materialize is a streaming database that uses incremental computation to maintain always-fresh, strongly consistent SQL views over data fro… | 100 | 6386 | active |
| Apache Hudi Apache Hudi is an open data lakehouse platform built on a high-performance open table format that brings database functionality like transa… | 96 | 6336 | stable |
| Pachyderm Pachyderm is a data-centric pipeline platform that automates data transformations with built-in data versioning and lineage tracking. It ru… | 30 | 6309 | active |
| apache/nifi Apache NiFi is an easy-to-use, powerful, and reliable system to process and distribute data, built on the JVM. It provides a browser-based … | 93 | 6249 | stable |
| apache/hive Apache Hive is a distributed, fault-tolerant data warehouse system that enables reading, writing, and managing petabytes of data in distrib… | 77 | 6030 | active |
| DataX Web DataX-Web is a distributed data synchronization tool built on top of Alibaba's DataX, providing a web UI to visually configure and manage d… | 23 | 6016 | stable |
| data load tool (dlt) dlt (data load tool) is an open-source Python library for building ELT data pipelines that extract data from REST APIs, SQL databases, clou… | 99 | 5955 | stable |
| lakeFS lakeFS is an open-source data version control service that turns object storage (S3, Azure Blob, GCS) into a Git-like repository with branc… | 93 | 5547 | stable |
| neo4j-labs/llm-graph-builder A web application that transforms unstructured data (PDFs, DOCs, TXTs, YouTube videos, web pages) into a knowledge graph stored in Neo4j us… | 88 | 5264 | active |
| fluvio-community/fluvio Fluvio is a distributed data streaming engine written in Rust that combines a Kafka-like event streaming platform with the Stateful DataFlo… | 79 | 5256 | active |
| Jitsu Jitsu is an open-source, self-hostable event data platform and Segment alternative that collects event data from websites, apps, and server… | 95 | 5108 | active |
| tidyverse/dplyr dplyr is an R package providing a consistent grammar of data manipulation with verbs like mutate(), select(), filter(), summarise(), and ar… | 75 | 5072 | stable |
| javascriptdata/danfojs Danfo.js is a JavaScript library providing high-performance, intuitive data structures (Series and DataFrame) for manipulating and processi… | 54 | 5063 | active |
| ArroyoSystems/arroyo Arroyo is a distributed stream processing engine written in Rust that lets users run stateful computations on high-volume real-time data st… | 76 | 5048 | active |
| clidey/whodb WhoDB is a lightweight, self-hosted database management workspace that lets teams explore schemas, edit data, run queries, and visualize re… | 86 | 5033 | active |
| deepseek-ai/smallpond Smallpond is a lightweight distributed data processing framework built on DuckDB and DeepSeek's 3FS shared file system. It lets users proce… | 21 | 5017 | active |
| dataset dataset is a Python library that makes reading and writing data in SQL databases as simple as working with JSON files. It provides implicit… | 72 | 4870 | stable |
| Chonkie Chonkie is a lightweight Python library for chunking text in RAG pipelines, offering multiple chunking algorithms, embeddings, and integrat… | 82 | 4790 | active |
| Amundsen Amundsen is an open-source data discovery and metadata engine that indexes data resources such as tables, dashboards, and streams, and powe… | 10 | 4782 | active |
| SPLWare/esProc esProc SPL is a JVM-based scripting language for structured data computation, usable as a standalone data analysis tool or embedded computi… | 77 | 4676 | active |
| dataabc/weibo-crawler A Python crawler for Sina Weibo that scrapes user profiles and posts, exporting data to CSV, JSON, MySQL, MongoDB, or SQLite, and optionall… | 70 | 4663 | active |
| yihong0618/running_page A self-hosted personal running homepage that syncs workout data from services like Strava, Garmin, Nike Run Club, and GPX files, then visua… | 77 | 4540 | active |
| RudderStack RudderStack's open-source event streaming server, a privacy- and security-focused Segment alternative written in Go. It collects customer e… | 96 | 4504 | active |
| unionai-oss/pandera Pandera is a Python library for statistical data validation of dataframe-like objects, supporting pandas, polars, PySpark, and others. It l… | 99 | 4481 | stable |
| rom1504/img2dataset A Python tool that downloads large sets of image URLs and packages them into machine learning datasets, with resizing and caption support. … | 52 | 4454 | active |
| tair-opensource/RedisShake RedisShake is a Go-based tool for migrating and transforming data between Valkey/Redis instances, supporting standalone, master-slave, sent… | 90 | 4443 | active |
page 1 / 7 next →