damklis/DataEngineeringProject
Example end to end data engineering project. observed · 2026-08-28
Health v2 · maintenance only
32/100
- Activity 0
- Release rhythm 35
- Longevity 100
Flags: no_releases
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-02. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 2255
- days_rel: n/a
- days_push: 1364
- n_releases_24m: 0
Adoption not part of the score
1429 stars · 278 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
An end-to-end data engineering project that scrapes news from RSS feeds via Airflow-scheduled Python scrapers and streams them through Kafka, MongoDB, Debezium, and Elasticsearch with MinIO for persistence. It exposes the collected news through a Django REST API, demonstrating a CQRS-style pipeline built entirely from open-source components.
Use cases
- build an end-to-end data pipeline with kafka and airflow
- scrape news from rss feeds into a searchable store
- learn how debezium cdc syncs mongodb to elasticsearch
- example of cqrs with mongodb and elasticsearch
- set up kafka connect sinks for mongo, elasticsearch, and s3
- serve scraped data through a rest api
- reference architecture for a data engineering portfolio project
When to choose
- you want a working reference implementation of a streaming data pipeline
- you're learning kafka, kafka connect, debezium, and airflow together
- you need a template for scraping plus serving data via an API
When to avoid
- you need a production-grade, actively maintained product rather than a demo
- your stack doesn't include kafka, mongodb, or elasticsearch
- you need a lightweight pipeline without heavy infrastructure
Facets
application · maturity maintenance
web-scraping streaming etl search-engine api-framework scheduling caching big-data crawlers apis python self-hosted kafka airflow debezium elasticsearch mongodb minio redis cqrs rss-feeds data-pipeline example-project data-engineering search automation docker web-server
1 source
- readme: https://github.com/damklis/DataEngineeringProject · fetched 2026-08-28 · 19dc6b792688
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| damklis/DataEngineeringProject | main | 32 |
For agents
markdown · JSON · MCP: product_card(name="damklis/DataEngineeringProject")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem