spotify/luigi
Luigi is a Python module that helps you build complex pipelines of batch jobs. It handles dependency resolution, workflow management, visualization etc. It also comes with Hadoop support built in. observed · 2026-08-28
Health v2 · maintenance only
91/100
- Activity 93
- Release rhythm 83
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-02. Adoption (stars, forks) is never an input.
- gap_med: 22
- age_days: 5095
- days_rel: 118
- days_push: 46
- n_releases_24m: 8
Adoption not part of the score
18765 stars · 2464 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
Luigi is a Python package for building complex pipelines of long-running batch jobs. It handles dependency resolution, workflow management, failure handling, and visualization, with built-in Hadoop, Spark, and Hive integrations.
Use cases
- orchestrate batch data pipelines in python
- resolve task dependencies in etl workflows
- schedule hadoop and spark jobs
- build multi-step data processing pipelines
- visualize and monitor long-running batch workflows
When to choose
- you need to chain many batch tasks with dependency resolution
- your pipelines include hadoop, spark, hive, or database jobs
- you want a mature, battle-tested python workflow library
When to avoid
- you need real-time or streaming data processing
- you prefer a modern dag-based orchestrator with a rich web UI like Airflow or Prefect
Facets
library · maturity stable
workflow-automation scheduling etl big-data python cli batch-jobs hadoop spark data-pipelines task-dependency-resolution data-engineering automation
2 sources
- readme: https://github.com/spotify/luigi · fetched 2026-08-28 · a9ad3d5e9cd7
- registry_pypi: https://pypi.org/pypi/luigi/json · fetched 2026-08-29 · 3092fc4bf226
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| spotify/luigi | main | 91 |
For agents
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem