Open-source etl software
673 open-source libraries, tools and applications tagged etl, ranked by how closely etl is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
673 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| neo4j-contrib/neo4j-apoc-procedures APOC (Awesome Procedures On Cypher) is a library of hundreds of procedures and functions that extend Neo4j's Cypher query language. It is p… | 89 | 1868 | active |
| pinterest/secor Secor is a Java service that persists Apache Kafka log messages to object stores such as Amazon S3, Google Cloud Storage, Azure Blob Storag… | 75 | 1858 | stable |
| davyxu/tabtoy Tabtoy is a high-performance command-line tool that exports spreadsheet data (Xlsx/CSV) into JSON, binary, and source code for Golang, C#, … | 23 | 1854 | active |
| dbt dbt-utils is an official dbt package providing reusable Jinja macros and generic data tests for dbt projects. It includes SQL generators (d… | 93 | 1822 | stable |
| buildship-ai/rowy Rowy is an open-source low-code backend platform for Firebase and Google Cloud that provides an Airtable-like spreadsheet UI for managing F… | 23 | 6847 | maintenance |
| AI Insight Daily A Cloudflare Workers-based content aggregation and generation platform that daily curates AI industry news, trending open-source projects, … | 64 | 1809 | active |
| apache/storm Apache Storm is a free and open-source distributed realtime computation system for processing unbounded streams of data, providing primitiv… | 97 | 6696 | maintenance |
| br-g/openf1 OpenF1 is a free, open-source API providing real-time and historical Formula 1 data, including lap timings, car telemetry, driver info, and… | 61 | 1779 | active |
| Bruin Bruin is an end-to-end data platform whose CLI combines data ingestion (via its ingestr tool), SQL/Python/R transformations, and data quali… | 91 | 1774 | active |
| lit26/finvizfinance A Python library that scrapes and downloads financial data from the FinViz website, returning stock fundamentals, technicals, charts, news,… | 85 | 1753 | active |
| bespokelabsai/curator Bespoke Curator is a Python library for bulk LLM inference and scalable synthetic data curation for post-training and structured data extra… | 89 | 1744 | active |
| lf-edge/ekuiper LF Edge eKuiper is a lightweight stream processing engine for IoT edge devices, offering SQL-based and graph-based rule engine for real-tim… | 98 | 1739 | active |
| jldbc/pybaseball pybaseball is a Python package that scrapes and retrieves current and historical baseball statistics from sources like MLB Statcast (Baseba… | 46 | 1735 | active |
| 4paradigm/OpenMLDB OpenMLDB is an open-source machine learning database that acts as a feature platform, computing consistent features for both offline traini… | 65 | 1714 | active |
| osm2pgsql-dev/osm2pgsql osm2pgsql is a command-line ETL tool that imports OpenStreetMap data in OSM, PBF, and O5M formats into a PostgreSQL/PostGIS database. It su… | 83 | 1685 | stable |
| Multiwoven/multiwoven Multiwoven is an open-source Reverse ETL and data activation platform that syncs data from warehouses like Snowflake, BigQuery, Redshift, a… | 88 | 1675 | active |
| sunlabuiuc/PyHealth PyHealth is an open-source Python toolkit for clinical deep learning, unifying healthcare datasets (EHRs, physiological signals, medical im… | 95 | 1673 | active |
| mortada/fredapi fredapi is a Python wrapper around the Federal Reserve Bank of St. Louis FRED and ALFRED web services for fetching economic data. It return… | 48 | 1658 | active |
| skrub-data/skrub skrub is a Python library that facilitates machine learning with dataframes, providing preprocessing, encoding, and wrangling tools for tab… | 92 | 1653 | active |
| huggingface/aisheets Hugging Face AI Sheets is an open-source no-code web application for building, enriching, and transforming datasets using AI models. It can… | 64 | 1652 | active |
| Snowflake-Labs/pg_lake pg_lake is a set of PostgreSQL extensions that turn Postgres into a lakehouse engine for Apache Iceberg tables and raw data lake files in o… | 82 | 1645 | active |
| karlicoss/HPI HPI (Human Programming Interface) is a Python package ('my') that unifies personal data from social networks, reading, health, location, me… | 94 | 1644 | active |
| meta-llama/synthetic-data-kit A Python CLI tool from Meta for generating high-quality synthetic datasets to fine-tune LLMs. It follows a four-stage pipeline (ingest, cre… | 39 | 1639 | active |
| ariacom/Seal-Report Seal Report & Task is an open-source (MIT) reporting and business intelligence framework written in C# for .NET, enabling users to build, p… | 89 | 1637 | active |
| OpenDota OpenDota is an open-source Dota 2 data platform that provides a REST API and web UI for match and player statistics. It ingests data from V… | 76 | 1632 | active |
| Snorkel Snorkel is a Python library for programmatically building and managing training data using weak supervision, letting users write labeling f… | 65 | 6010 | maintenance |
| nerevu/riko riko is a pure Python stream processing library modeled after Yahoo! Pipes, combining reusable, configuration-driven modular pipes with syn… | 96 | 1607 | active |
| cuducos/minha-receita Minha Receita is an open-source project that consolidates Brazilian Federal Revenue (Receita Federal) CNPJ company data from scattered CSV … | 10 | 1603 | active |
| paradigmxyz/cryo cryo is a Rust-based CLI tool (also available as a Python package) that extracts Ethereum/EVM blockchain data into parquet, csv, , or Pytho… | 20 | 1581 | active |
| getdozer/dozer Dozer is a real-time data movement tool written in Rust that captures change data (CDC) from sources like Postgres, MySQL, Snowflake, and K… | 23 | 1578 | active |
| quixio/quix-streams Quix Streams is a pure Python framework for building real-time data pipelines and event-driven applications on Apache Kafka using a Streami… | 96 | 1574 | active |
| dimitri/pgcopydb pgcopydb is a command-line tool that automates copying a PostgreSQL database between two running servers, combining pg_dump/pg_restore for … | 79 | 1567 | active |
| mosaicml/streaming StreamingDataset is a Python library from MosaicML for fast, accurate streaming of training data from cloud object storage (S3, GCS, Azure,… | 66 | 1560 | active |
| ilyakatz/data-migrate A Ruby gem for Rails that lets you run data migrations alongside schema migrations. Data migrations live in db/data and are tracked in a da… | 66 | 1555 | active |
| FinanceData/FinanceDataReader FinanceDataReader is a Python library and CLI tool for reading financial data such as stock listings, stock prices, indexes, exchange rates… | 66 | 1555 | active |
| Legend Legend is an end-to-end data platform from FINOS (originated at Goldman Sachs) covering the full data lifecycle: modeling, curation, queryi… | 77 | 1552 | active |
| combust/mleap MLeap is a serialization format (Bundle.ML) and portable execution engine for machine learning pipelines, implemented in Scala with Python … | 98 | 1546 | active |
| tensorflow/gnn TensorFlow GNN is a Python library for building Graph Neural Networks on TensorFlow, including a GraphTensor type for heterogeneous graphs,… | 66 | 1546 | active |
| uwdata/arquero Arquero is a JavaScript library for query processing and transformation of array-backed, column-oriented data tables. It offers a fluent, d… | 41 | 1542 | stable |
| BemiHQ/BemiDB BemiDB is an open-source analytical data warehouse that combines built-in data source connectors (like Fivetran) with a Postgres-compatible… | 50 | 1536 | active |
| Bumblebee Optimus is a Python library for agile data preparation that provides a unified API over pandas, Dask, cuDF, Dask-cuDF, Vaex, and PySpark. I… | 23 | 1535 | active |
| pyper-dev/pyper Pyper is a pure-Python library for building concurrent and parallel data pipelines using functional programming patterns. It unifies thread… | 26 | 1517 | active |
| apache/inlong Apache InLong is a one-stop, full-scenario integration framework for massive data, supporting data ingestion, synchronization, and subscrip… | 86 | 1503 | stable |
| lqzhgood/Shmily Shmily is a large umbrella project for exporting and archiving personal chat and communication records from QQ, WeChat, SMS, call logs, pho… | 51 | 1503 | active |
| pyjanitor-devs/pyjanitor pyjanitor is a Python library providing clean, readable APIs for data cleaning on pandas DataFrames, inspired by the R package janitor. It … | 96 | 1500 | active |
| Vincentqyw/cv-arxiv-daily An automated daily digest of computer vision and robotics arXiv papers (SLAM, SFM, visual localization, keypoint detection, image matching,… | 76 | 1500 | active |
| apache/hop Apache Hop is an open-source data and metadata orchestration platform for visually designing and running data integration pipelines and wor… | 91 | 1488 | active |
| datazip-inc/olake OLake Go is a high-performance open-source EL (extract-load) engine written in Go that replicates databases (PostgreSQL, MySQL, MongoDB, Or… | 85 | 1474 | active |
| Surfer-Org/Protocol Surfer Protocol is an open-source framework for exporting personal data from platforms like Gmail, iMessages, Twitter, Notion, and ChatGPT.… | 14 | 1474 | active |
| sfirke/janitor janitor is an R package with simple, user-friendly functions for examining and cleaning dirty data, such as formatting data.frame column na… | 23 | 1458 | stable |
| cube2222/octosql OctoSQL is a Go CLI tool that lets you query, join, and transform data from multiple databases and file formats using SQL through a unified… | 23 | 5268 | maintenance |
| dadoonet/fscrawler FSCrawler is a Java-based file system crawler that indexes binary documents (PDF, MS Office, Open Office) into Elasticsearch, tracking new,… | 86 | 1452 | active |
| wgzhao/Addax Addax is a fast, extensible ETL tool for synchronizing data between heterogeneous SQL and NoSQL data sources, forked and evolved from Aliba… | 100 | 1448 | active |
| tidyverse/tidyr tidyr is an R package from the tidyverse that provides tools for reshaping messy data into tidy form, including pivoting between long and w… | 63 | 1442 | stable |
| winedarksea/AutoTS AutoTS is a Python library for automated time series forecasting, offering dozens of sklearn-style models (statistical, ML, deep learning) … | 89 | 1429 | active |
| datavane/tis TIS is an AI-native data integration platform built on DataX, Flink, and Flink-CDC that provides a visual Web-UI for zero-code batch and re… | 88 | 1424 | active |
| toluaina/pgsync PGSync is an open-source (MIT) change data capture tool that syncs PostgreSQL, MySQL, or MariaDB data to Elasticsearch or OpenSearch in rea… | 95 | 1423 | active |
| MLOps Python Package A Python package template that provides a production-grade code base with MLOps best practices for building and deploying machine learning … | 92 | 1419 | active |
| amphi-ai/amphi-etl Amphi is a visual data preparation and ETL tool that lets users build data pipelines through a drag-and-drop interface while generating sta… | 67 | 1411 | active |
| thinh-vu/vnstock Vnstock is an open-source Python library for extracting and analyzing Vietnam stock market data, returning data as pandas DataFrames via si… | 93 | 1408 | active |
| SebastienZh/StockTradebyZ A semi-automated stock screening application for China's A-share market that fetches daily K-line data via Tushare, applies quantitative pr… | 48 | 1395 | active |
| Data-Forge Data-Forge is a TypeScript data transformation and analysis toolkit for JavaScript, inspired by Pandas and LINQ. It provides a DataFrame/Se… | 65 | 1394 | active |
| okfn-brasil/querido-diario Querido Diário is an open-source project by Open Knowledge Brasil that scrapes and aggregates Brazilian municipal official gazettes (diário… | 75 | 1391 | active |
| Mirobody Mirobody is an open-source, AI-native health data engine that collects readings from lab reports, wearables, and genomics, standardizes the… | 84 | 1385 | active |
| locationtech/geotrellis GeoTrellis is a Scala library and framework for high-performance reading, writing, and processing of geospatial raster and vector data. It … | 82 | 1377 | stable |
| spatie/simple-excel A PHP library for reading and writing simple Excel (xlsx) and CSV files with a fluent API. It uses generators and LazyCollections to keep m… | 80 | 1372 | active |
| nf-core/rnaseq nf-core/rnaseq is a Nextflow-based bioinformatics pipeline for analyzing RNA sequencing data, performing QC, trimming, (pseudo-)alignment w… | 85 | 1371 | active |
| duckdb/dbt-duckdb dbt-duckdb is the dbt adapter plugin for DuckDB, an embedded OLAP database. It lets you run dbt data transformation pipelines in SQL or Pyt… | 91 | 1366 | active |
| nextstrain/ncov A Nextstrain build pipeline that analyzes SARS-CoV-2 viral genomes to understand their evolution and spread, producing phylogenetic visuali… | 82 | 1363 | active |
| SpiderClub/weibospider A distributed web crawler for Sina Weibo (Chinese microblogging platform) built with Python, Celery, and requests. It scrapes user profiles… | 32 | 4790 | maintenance |
| subsquid/squid-sdk Squid SDK is a TypeScript ETL toolkit for building blockchain data indexers, supporting Ethereum-like chains, Substrate, and Solana, with d… | 96 | 1341 | active |
| slothflowlabs/duckle Duckle is an open-source ETL/ELT platform built on DuckDB that you self-host on your own servers or cloud, with a visual no-code/low-code c… | 79 | 1336 | active |
| rwynn/monstache Monstache is a Go daemon that continuously syncs MongoDB collections into Elasticsearch (or OpenSearch) in realtime using change streams. I… | 43 | 1332 | active |
| petl-developers/petl petl is a general-purpose Python package for extracting, transforming and loading tables of data. It provides a lightweight, pure-Python to… | 97 | 1318 | stable |
| GoogleCloudPlatform/DataflowTemplates A collection of Google-provided Apache Beam pipeline templates for Google Cloud Dataflow that solve common in-cloud data tasks like import/… | 95 | 1317 | active |
| Mojang/DataFixerUpper DataFixerUpper is a Java library from Mojang for incrementally building, merging, and optimizing data transformations between schema versio… | 55 | 1313 | active |
| arkflow-rs/arkflow ArkFlow is a high-performance stream processing engine written in Rust on top of Tokio, connecting configurable inputs (Kafka, MQTT, HTTP, … | 68 | 1306 | active |
| apache/incubator-xtable Apache XTable (incubating) is a cross-table converter that translates lakehouse table format metadata between Apache Hudi, Apache Iceberg, … | 83 | 1302 | active |
| elixir-explorer/explorer Explorer is an Elixir library providing series (one-dimensional) and dataframes (two-dimensional) for fast data exploration and manipulatio… | 93 | 1296 | active |
| apache/datafusion-comet Apache DataFusion Comet is a high-performance accelerator plugin for Apache Spark that keeps queries Arrow-native end-to-end, executing ope… | 90 | 1295 | active |
| 0xSero/ai-data-extraction A Python toolkit of extraction scripts that pulls complete chat, agent, and code-context history from AI coding assistants like Cursor, Cla… | 60 | 1291 | active |
| DTStack/Taier Taier is a self-hosted distributed dispatching platform for big data that handles task submission, DAG-based scheduling, and operations/mai… | 78 | 1284 | active |
| Nixtla/mlforecast mlforecast is a Python framework for time series forecasting using any machine learning model with fit/predict methods, providing efficient… | 94 | 1282 | active |
| firecrawl/fire-enrich Fire Enrich is an AI-powered data enrichment web application that transforms a list of email addresses into rich company datasets, includin… | 36 | 1277 | active |
| astronomer/astronomer-cosmos Astronomer Cosmos is an open-source Python library that converts dbt Core or dbt Fusion projects into Apache Airflow DAGs and task groups w… | 96 | 1276 | active |
| meta-pytorch/data TorchData is a PyTorch library providing scalable, performant data loading utilities, including StatefulDataLoader, a drop-in replacement f… | 73 | 1267 | active |
| zinggAI/zingg Zingg is an ML-based tool for scalable master data management, entity resolution, identity resolution, and record deduplication. It runs on… | 83 | 1253 | active |
| sentinel-hub/eo-learn eo-learn is a collection of open-source Python packages for accessing and processing spatio-temporal satellite imagery, built around modula… | 65 | 1251 | active |
| darold/ora2pg Ora2Pg is a free Perl-based tool that migrates an Oracle database to PostgreSQL by scanning the source database and generating SQL scripts … | 66 | 1238 | active |
| kevwan/go-stash go-stash is a high-performance, open-source server-side data processing pipeline written in Go that ingests data from Kafka, applies config… | 54 | 1225 | active |
| marcboeker/gmail-to-sqlite A Python CLI application that syncs Gmail messages into a local SQLite database, supporting incremental and full syncs with deletion detect… | 71 | 1223 | active |
| skyplane-project/skyplane Skyplane is a CLI tool for blazingly fast bulk data transfers between cloud object stores (AWS S3, Azure Blob, GCS, IBM COS) and local disk… | 23 | 1222 | active |
| pytroll/satpy Satpy is a Python library for reading, manipulating, and writing meteorological remote sensing data from earth-observing satellites. It sup… | 82 | 1209 | active |
| Mirth Connect Mirth Connect by NextGen Healthcare is an open-source healthcare integration engine that filters, transforms, extracts, and routes messages… | 66 | 1203 | active |
| gityuanbao/share A personal open-source repository whose main component is akshare_collector, a Python tool built on AKShare that collects Chinese financial… | 61 | 1200 | active |
| xorbitsai/xorbits Xorbits is an open-source distributed computing framework that scales Python data science and machine learning workloads from a laptop to l… | 61 | 1200 | active |
| wireservice/agate agate is a Python data analysis library optimized for humans instead of machines, offering a readable alternative to numpy and pandas. It p… | 75 | 1198 | stable |
| xataio/pgstream pgstream is an open-source change data capture (CDC) tool and Go library that replicates PostgreSQL data, including DDL/schema changes, to … | 85 | 1195 | active |
| marsupialtail/quokka Quokka is a lightweight distributed dataflow/query engine written in Python, built on Ray, DuckDB, Polars, and Arrow, designed for stateful… | 23 | 1193 | active |
| apache/amoro Apache Amoro (incubating) is a Lakehouse management system built on open data lake formats like Iceberg, Paimon, and Mixed-Hive. It provide… | 71 | 1191 | active |