apache/nutch
Apache Nutch is an extensible and scalable web crawler observed · 2026-08-28
Health v2 · maintenance only
77/100
- Activity 99
- Release rhythm 35
- Longevity 100
Flags: no_releases
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 6314
- days_rel: n/a
- days_push: 8
- n_releases_24m: 0
Adoption not part of the score
3277 stars · 1279 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
Apache Nutch is a highly extensible and scalable open-source web crawler built on Apache Hadoop data structures. It supports batch crawling of large web volumes with a pluggable architecture for parsing (Tika) and indexing (Solr, Elasticsearch).
Use cases
- crawl large portions of the web for a search engine index
- batch-scrape millions of pages into Hadoop
- collect and parse web content for downstream data pipelines
- build a custom crawler with pluggable parsing and indexing
- feed documents into Solr or Elasticsearch from web sources
When to choose
- you need large-scale, Hadoop-based batch crawling
- you want a mature, production-ready crawler with plugin extensibility
- you need fine-grained control over parsing, filtering, scoring, and indexing
When to avoid
- you need a lightweight single-machine crawler for a few sites
- you need real-time or low-latency crawling
- you want a crawler with built-in authentication for its REST API (the legacy service has none)
Facets
application · maturity stable
web-scraping parser search-engine etl plugin-system crawlers big-data developer-tools jvm self-hosted web-crawler hadoop apache batch-crawling tika solr elasticsearch search linux docker
4 sources
- readme: https://github.com/apache/nutch · fetched 2026-08-28 · ab4abfe52a6e
- homepage: https://nutch.apache.org/ · fetched 2026-08-29 · 3b433f1c896f
- site_page: https://nutch.apache.org/documentation/security · fetched 2026-08-29 · f3d35b2f185f
- site_page: https://nutch.apache.org/documentation · fetched 2026-08-29 · b6909a976ae2
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| apache/nutch | main | 77 |
For agents
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem