Open-source etl software
673 open-source libraries, tools and applications tagged etl, ranked by how closely etl is their primary purpose, then by adoption and maintenance health. Each entry links to a page with sources, use cases and when to choose or avoid it.
673 products, primary matches first, then adoption-weighted; health v2 shown.
| Product | Health v2 | Stars | Maturity |
|---|---|---|---|
| Apache Camel Apache Camel is an open-source integration framework offering 350+ connectors for databases, APIs, message brokers, and cloud services, wit… | 77 | 6371 | stable |
| Open-Meteo Open-Meteo is an open-source weather API service that aggregates numerical weather prediction models from 15+ national weather services int… | 85 | 6244 | active |
| apache/pinot Apache Pinot is an open-source distributed OLAP datastore purpose-built for low-latency, high-throughput real-time analytics. It ingests da… | 82 | 6163 | stable |
| MontFerret/ferret Ferret is a declarative, expression-oriented query language (FQL) with an embeddable Go runtime for querying, transforming, and automating … | 79 | 6013 | active |
| AidLux (AidLearning) AidLux (originally AidLearning) is an AIoT development platform that runs a native Ubuntu Linux environment with GUI, deep learning tooling… | 63 | 5812 | active |
| Eventual-Inc/Daft Daft is a high-performance distributed data engine with a Python dataframe API, implemented in Rust, designed for AI and multimodal workloa… | 95 | 5804 | active |
| NVIDIA/DALI NVIDIA DALI is a GPU-accelerated data loading and preprocessing library with optimized building blocks and an execution engine for deep lea… | 91 | 5761 | active |
| Nuclio Nuclio is a high-performance open-source serverless (FaaS) platform for real-time event and data processing, deployable standalone via Dock… | 92 | 5760 | stable |
| ZenML ZenML is an open-source MLOps framework for building, deploying, and managing production ML pipelines, LLM workflows, and AI agents on any … | 97 | 5601 | active |
| superduper-io/superduper Superduper is an open-source Python framework for building database-integrated AI applications and agents, connecting models, embeddings, a… | 44 | 5329 | active |
| microsoft/SynapseML SynapseML (formerly MMLSpark) is an open-source machine learning library built on Apache Spark that provides simple, composable, distribute… | 84 | 5247 | active |
| apache/iggy Apache Iggy is a high-performance, persistent message streaming platform written in Rust, supporting TCP, QUIC, HTTP, and WebSocket transpo… | 92 | 5035 | active |
| OpenTelemetry Collector The contrib repository for the OpenTelemetry Collector, containing 200+ vendor-neutral components (receivers, processors, exporters) for re… | 100 | 5009 | active |
| Zep A repository of example code, framework integration packages, ingestion tooling, and memory benchmarks for Zep Cloud, a managed agent memor… | 97 | 4959 | active |
| apache/rocketmq-externals The Apache RocketMQ externals repository is the community home for incubating ecosystem projects around the RocketMQ message broker, such a… | 74 | 4593 | active |
| medialab/xan xan is a fast, memory-efficient Rust command line tool for processing, transforming, and exploring CSV/TSV files directly from the shell. I… | 96 | 4517 | active |
| dedupe Dedupe is a Python library that uses machine learning to perform fuzzy matching, deduplication, and entity resolution on structured data. I… | 44 | 4516 | stable |
| salesforce/Merlion Merlion is a Python library from Salesforce providing an end-to-end machine learning framework for time series intelligence, covering forec… | 10 | 4486 | active |
| mosaicml/llm-foundry LLM Foundry is a PyTorch-based codebase for training, finetuning, evaluating, and deploying large language models from 125M to 70B+ paramet… | 62 | 4449 | active |
| zvtvz/zvt ZVT is a unified, modular quantitative trading framework written in Python for stocks and cryptocurrencies. It provides data ingestion, fac… | 73 | 4311 | active |
| kubeflow/pipelines Kubeflow Pipelines is a Kubernetes-native platform for building, deploying, and orchestrating reusable end-to-end machine learning workflow… | 98 | 4248 | stable |
| facebookincubator/velox Velox is a composable, extensible C++ execution engine library for data management systems, created by Meta. It provides reusable vectorize… | 77 | 4233 | active |
| cartography-cncf/cartography Cartography is a Python CLI tool that ingests infrastructure assets and their relationships from 30+ platforms (AWS, GCP, Azure, Kubernetes… | 100 | 4147 | active |
| apache/tika Apache Tika is a Java toolkit that detects file types and extracts text and metadata from over a thousand file formats (PDF, Office documen… | 77 | 4100 | stable |
| RavenDB RavenDB is an open-source (AGPLv3) ACID NoSQL document database written in C#, featuring full-text search, vector search, time series, shar… | 92 | 4000 | stable |
| securo-finance/securo Securo is an open-source, self-hosted personal finance manager built with FastAPI and React that tracks accounts, transactions, budgets, go… | 76 | 3839 | active |
| DataLinkDC/dinky Dinky is an open-source, one-stop real-time computing platform built around Apache Flink, providing an IDE for FlinkSQL development, online… | 74 | 3769 | active |
| morphik-org/morphik-core Morphik Core is an open-source, source-available multimodal retrieval engine for building RAG applications over unstructured data like PDFs… | 65 | 3711 | active |
| TimelyDataflow/timely-dataflow A modular Rust implementation of the timely dataflow computational model from the Naiad paper, providing a low-latency cyclic data-parallel… | 89 | 3649 | active |
| apache/arrow-rs The official Rust implementation of Apache Arrow, providing the in-memory columnar format, and of Apache Parquet for columnar file reading … | 100 | 3641 | active |
| Docta-ai/docta Docta is a Python library for data-centric AI that diagnoses and fixes issues in datasets such as label errors, noise, and misannotations. … | 66 | 3479 | active |
| piskvorky/smart_open A Python library for efficiently streaming very large files to and from remote storages like S3, GCS, Azure Blob Storage, HDFS, WebHDFS, HT… | 92 | 3459 | stable |
| apache/nutch Apache Nutch is a highly extensible and scalable open-source web crawler built on Apache Hadoop data structures. It supports batch crawling… | 77 | 3297 | stable |
| Vortex Vortex is an extensible, high-performance columnar file format and toolkit for compressed data, positioned as a faster alternative to Apach… | 88 | 3260 | active |
| Barabama/FreeNodes A Python crawler that aggregates free proxy nodes (v2ray, Clash, vmess, vless, trojan, ss) from public websites and publishes them as auto-… | 73 | 3238 | active |
| Apache HugeGraph Apache HugeGraph is a fast, highly scalable graph database supporting tens of billions of vertices and edges with OLTP real-time queries vi… | 70 | 3192 | active |
| spiceai/spiceai Spice.ai OSS is a portable Rust runtime (~140 MB binary) that combines federated SQL querying, accelerated data materialization, full-text … | 94 | 3101 | active |
| deepnote/deepnote Deepnote is an open-source, AI-first drop-in replacement for Jupyter notebooks with a human-readable YAML format, block-based architecture … | 80 | 3017 | active |
| hosseinmoein/DataFrame A high-performance, templatized C++ library providing a heterogeneous DataFrame container for in-memory data exploration, transformation, a… | 95 | 2990 | active |
| NVIDIA/NeMo-Retriever NVIDIA's scalable document content and metadata extraction library (also known as NVIDIA Ingest) that splits documents, classifies and extr… | 82 | 2978 | active |
| google/osv.dev OSV (Open Source Vulnerabilities) is Google's open, distributed vulnerability database and triage service that aggregates security advisori… | 91 | 2973 | stable |
| rilldata/rill Rill is an open-source business intelligence tool that combines a local developer application and cloud platform for building interactive d… | 93 | 2915 | active |
| OpenDCAI/DataFlex DataFlex is a data-centric training framework built on LLaMA-Factory that dynamically schedules LLM training data during optimization. It p… | 63 | 2824 | active |
| Broadway Broadway is an Elixir library for building concurrent, multi-stage data ingestion and processing pipelines on top of GenStage. It provides … | 75 | 2689 | stable |
| nanomq/nanomq NanoMQ is an ultra-lightweight, high-performance MQTT broker and messaging bus written in C for IoT/IIoT edge computing and software-define… | 95 | 2619 | active |
| data-infra/cube-studio CubeStudio is an open-source, cloud-native, all-in-one AI platform covering the full machine learning lifecycle (MLOps/MaaS/LLMOps), includ… | 75 | 2546 | active |
| iliane5/meridian Meridian is an open-source AI-powered news intelligence system that scrapes hundreds of RSS sources, clusters and analyzes articles with LL… | 28 | 2437 | active |
| Elementary Elementary OSS is an open-source CLI for dbt-native data observability that reads warehouse metadata, dbt artifacts, and test results to ge… | 99 | 2421 | active |
| learnhouse/learnhouse LearnHouse is a next-generation open-source learning management system (LMS) for creating, sharing, and selling educational content. It com… | 100 | 2404 | active |
| syslog-ng/syslog-ng syslog-ng is an enhanced syslog daemon that collects, parses, filters, and routes log messages from a wide range of sources including syslo… | 88 | 2377 | stable |
| stephengpope/no-code-architects-toolkit A self-hostable Flask-based API that consolidates common media processing tasks—video editing, captioning, audio conversion, transcription,… | 47 | 2345 | active |
| refuel-ai/autolabel Autolabel is a Python library for labeling, cleaning, and enriching text datasets using large language models like GPT-4, Claude, and open-… | 22 | 2330 | active |
| dataabc/weibo-search A Python/Scrapy-based crawler that continuously fetches Weibo keyword and hashtag search results, including full post metadata, images, and… | 67 | 2328 | active |
| Nethereum Nethereum is the .NET integration platform for Ethereum and EVM-compatible blockchains, offering 130+ NuGet packages covering RPC access, s… | 82 | 2260 | active |
| galilai-group/stable-worldmodel A Python library providing a unified platform for reproducible world model research, covering data collection, training, and evaluation via… | 78 | 2260 | active |
| invoice-x/invoice2data A Python library and CLI tool that extracts structured data from PDF invoices using pluggable text-extraction backends (including OCR) and … | 96 | 2235 | active |
| op7418/Youtube-clipper-skill A Claude Code skill that downloads YouTube videos and uses AI to generate semantic chapters, clip segments, translate subtitles into biling… | 41 | 2217 | active |
| Apache Fluss Apache Fluss is a lakehouse-native streaming storage built for real-time analytics and AI, serving as the real-time data layer for Lakehous… | 80 | 2216 | active |
| RubixML/ML Rubix ML is a high-level open-source machine learning and deep learning library for PHP, offering 40+ supervised and unsupervised algorithm… | 98 | 2208 | active |
| OdyseeTeam/chainquery Chainquery is a Go daemon and API server that parses the LBRY blockchain into a structured MySQL database, syncing blocks, transactions, ad… | 79 | 2205 | active |
| alibaba/loongcollector LoongCollector (formerly iLogtail) is a high-performance, lightweight observability data collection agent from Alibaba for cloud-native env… | 89 | 2202 | active |
| thingsboard/thingsboard-gateway An open-source Python IoT gateway that integrates legacy and third-party devices with the ThingsBoard IoT platform. It acts as a protocol a… | 96 | 2192 | active |
| Vaex Vaex is a high-performance Python DataFrame library for lazy, out-of-core processing of large tabular datasets, using memory mapping and ze… | 53 | 8512 | maintenance |
| simonw/sqlite-utils sqlite-utils is a Python CLI utility and library for creating, querying, and manipulating SQLite databases. It supports piping JSON/CSV/TSV… | 96 | 2179 | stable |
| agentset-ai/agentset Agentset is an open-source RAG-as-a-service platform that handles document ingestion, chunking, embeddings, retrieval, and agentic search w… | 64 | 2103 | active |
| pydoit/doit doit is a Python-native task management and automation tool similar to make, where tasks are defined as plain Python functions and dicts in… | 75 | 2087 | active |
| karpathy/jobs A research tool that scrapes the Bureau of Labor Statistics Occupational Outlook Handbook (342 occupations) and renders an interactive tree… | 44 | 2029 | active |
| Mooncake-Labs/pg_mooncake pg_mooncake is a Postgres extension that maintains a columnstore mirror of Postgres tables in Apache Iceberg, enabling fast real-time analy… | 55 | 2003 | active |
| featureform/featureform Featureform is a virtual feature store that sits atop your existing data infrastructure and orchestrates it to define, manage, and serve ML… | 32 | 1991 | active |
| KEV0143/Parser-Chitai-Gorod A Python-based scraper for the Russian online bookstore Chitai-Gorod that collects book URLs across catalog pages and extracts structured p… | 72 | 1980 | active |
| faust-streaming/faust Faust-streaming is a Python stream processing library that ports Kafka Streams ideas to Python using asyncio, letting developers build dist… | 96 | 1889 | active |
| e-p-armstrong/augmentoolkit Augmentoolkit is a Python application that generates domain-expert fine-tuning datasets from uploaded documents and trains custom LLMs on t… | 65 | 1872 | active |
| Galaxy Galaxy is an open-source, web-based platform for accessible, reproducible, and transparent data-intensive scientific research, primarily in… | 92 | 1869 | stable |
| Datus-ai/Datus-agent Datus is an open-source (Apache-2.0) AI data engineering agent delivered primarily as a Python CLI, with a VS Code/Cursor extension and opt… | 85 | 1845 | active |
| microsoft/CDM Microsoft's Common Data Model (CDM) is a standard, extensible collection of schemas (entities, attributes, relationships) representing busi… | 32 | 1825 | active |
| alibaba/MongoShake MongoShake is a Go-based universal data replication platform built on MongoDB's oplog, fetching changes from a source MongoDB and replaying… | 92 | 1820 | active |
| NVIDIA-NeMo/Curator NVIDIA NeMo Curator is a scalable, GPU-accelerated toolkit for preprocessing and curating large text, image, video, and audio datasets for … | 88 | 1814 | active |
| DataArcTech/DataArc-SynData-Toolkit DataArc SynData Toolkit is a Python-based synthetic data generation platform for creating customized LLM training data from local corpora, … | 59 | 1783 | active |
| yobix-ai/extractous Extractous is a fast Rust library for extracting content and metadata from unstructured documents like PDF, Word, Excel, HTML, CSV, and ema… | 25 | 1780 | active |
| crazyykhllc-bit/CyberPPT CyberPPT is a Codex Skill that converts documents, research materials, and business data into high-density, editable, consulting-style Powe… | 52 | 1770 | active |
| T5 The official T5 library from Google Research, implementing the text-to-text transfer transformer for NLP tasks like summarization, question… | 67 | 6562 | maintenance |
| philhagen/sof-elk SOF-ELK is a pre-built virtual appliance based on the Elastic stack (Elasticsearch, Logstash, Kibana, Filebeat) tailored for computer foren… | 75 | 1755 | active |
| kyleskom/NBA-Machine-Learning-Sports-Betting A Python application that predicts NBA game winners and totals (over/under) using XGBoost and neural network models trained on historical t… | 78 | 1747 | active |
| datageartech/datagear DataGear is an open-source, self-hosted data visualization and analysis platform for freely building interactive data dashboards. It connec… | 77 | 1740 | active |
| lotus-data/lotus LOTUS is a Python library providing a Pandas-like API of LLM-powered semantic operators (map, filter, extract, aggregate, top-k) for bulk p… | 85 | 1719 | active |
| MLRun MLRun is an open-source MLOps and AI orchestration framework for building, training, deploying, and monitoring machine learning and generat… | 92 | 1697 | active |
| matanolabs/matano Matano is an open-source, cloud-native security data lake that runs in your AWS account, normalizing unstructured security logs into a stru… | 23 | 1696 | active |
| drmingler/docling-api A self-hostable FastAPI backend service that converts documents (PDF, DOCX, PPTX, HTML, images, CSV, AsciiDoc, Markdown) into Markdown usin… | 62 | 1693 | active |
| reata/sqllineage SQLLineage is a Python library and CLI tool that analyzes SQL statements to extract source and target table lineage without requiring users… | 88 | 1677 | active |
| deepsense-ai/ragbits Ragbits is a Python framework of modular building blocks for rapidly developing generative AI applications, covering LLM interaction, promp… | 69 | 1674 | active |
| HumanSignal/Adala Adala is a Python framework for building autonomous data labeling agents powered by LLMs. Agents learn skills iteratively from a ground tru… | 63 | 1644 | active |
| pixeltable/pixeltable Pixeltable is a Python library providing declarative, incremental data infrastructure for multimodal AI applications, unifying storage of i… | 94 | 1639 | active |
| rulego/rulego RuleGo is a lightweight, high-performance, embedded rule engine and component orchestration framework for Go. It lets developers compose re… | 88 | 1621 | active |
| guacsec/guac GUAC (Graph for Understanding Artifact Composition) is an OpenSSF incubating project that ingests software security metadata such as SBOMs,… | 88 | 1557 | active |
| emcf/thepipe thepipe is a Python package and API that extracts clean markdown, multimodal media, and structured data from tricky documents like PDFs, Wo… | 80 | 1529 | active |
| Open-Source-Legal/OpenContracts OpenContracts is a self-hosted, MIT-licensed document intelligence platform that turns document repositories into a programmable citation g… | 90 | 1508 | active |
| locationtech/geomesa GeoMesa is an open-source suite of JVM-based tools for large-scale geospatial querying and analytics on distributed computing systems. It p… | 76 | 1499 | active |
| AI4Finance-Foundation/FinNLP FinNLP is a Python library for collecting internet-scale financial data from sources like Finnhub, Yahoo Finance, Reuters, and Sina Finance… | 31 | 1491 | active |
| shuttle-hq/synth Synth is a declarative data generator written in Rust that produces realistic synthetic test data from a version-controllable data model. I… | 23 | 1486 | active |
| refinery An open-source tool for scaling, assessing, and maintaining natural language training data, treating datasets like software artifacts. It p… | 23 | 1473 | active |