# apache/gobblin

A distributed data integration framework that simplifies common aspects of big data integration such as data ingestion, replication, organization and lifecycle management for both streaming and batch data ecosystems.

Repository: https://github.com/apache/gobblin
Canonical: https://ross.abutalabs.com/products/gobblin
Homepage: https://gobblin.apache.org/
Language: Java
License: Apache-2.0
License Family: permissive
Topics: data, ingestion, replication, management, apache
Last push: 2026-07-31T06:45:24+00:00

## Health v2 (maintenance only)
Score: 66/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 95, release rhythm 8, longevity 100
- inputs: {"age_days": 4293, "days_push": 33, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2270, forks 750 (observed 2026-08-28T04:06:33.103586+00:00)

## What it is
Apache Gobblin is a distributed data integration framework for ingesting, replicating, organizing, and managing lifecycle of data across streaming and batch ecosystems. It runs at petabyte scale in production and supports ELT patterns with sources and sinks like Kafka, HDFS, S3, ADLS, and vendor APIs.

## Use cases
- ingest kafka topics into a data lake on hdfs or s3
- replicate data between hdfs and s3 or adls
- bulk load a serving store like couchbase from the data lake
- integrate salesforce api data into a data warehouse
- enforce data retention and gdpr deletion policies
- schedule and orchestrate batch and streaming ingestion jobs
- compact and deduplicate datasets in a data lake

## When to choose
- you need scalable, fault-tolerant data ingestion into a data lake
- you want incremental processing with state management and atomic publishing
- you need both streaming and batch ingestion in one framework
- you operate at large scale and need battle-tested infrastructure

## When to avoid
- you need general-purpose data transformation like Spark or Flink
- you need a general workflow orchestrator like Airflow or Dagster
- you need a data storage system
- your use case is small-scale simple ETL where a lighter tool suffices

## Facets
- artifact type: framework
- maturity: stable
- function: etl, streaming, workflow-automation, scheduling, data-science
- domain: big-data, microservices
- platform: jvm, cloud, self-hosted
- tags: data-ingestion, data-lake, elt, kafka, hadoop, data-replication, data-retention, gdpr-compliance, apache-project, data-engineering, automation, linux, docker

## Member repositories
- apache/gobblin (main) score 66

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:06:33.103586+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T02:41:49.487570+00:00, confidence not recorded.
  - readme: https://github.com/apache/gobblin (fetched 2026-08-28T04:06:33.103586+00:00, sha 48d2b89aeff1)
  - homepage: https://gobblin.apache.org/ (fetched 2026-08-29T10:22:18.267159+00:00, sha 50154bdb5505)
  - site_page: https://gobblin.apache.org/docs/developer-guide/HighLevelConsumer (fetched 2026-08-29T10:22:18.276210+00:00, sha 8b14cd766b74)
- Data as of 2026-08-30T08:39:29.467469+00:00.