internetarchive/heritrix3
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. observed · 2026-08-28
Health v2 · maintenance only
95/100
- Activity 99
- Release rhythm 87
- Longevity 100
Flags: no_license
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: 41.5
- age_days: 5430
- days_rel: 9
- days_push: 9
- n_releases_24m: 17
Adoption not part of the score
3305 stars · 792 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler written in Java. It crawls websites and stores content in WARC archives while respecting robots.txt and politeness policies.
Use cases
- crawl entire websites for archival purposes
- build web archive collections in WARC format
- mirror large portions of the web at scale
- respectfully crawl sites honoring robots.txt and politeness settings
- run scheduled recurring crawls of web resources
- extend crawler behavior with custom Java modules
When to choose
- you need archival-quality, standards-compliant web crawling with WARC output
- you want a mature, battle-tested crawler used by the Internet Archive
- you need fine-grained politeness and robots.txt compliance
- you want an extensible crawler with a REST API and bean-based configuration
When to avoid
- you only need lightweight scraping of a few pages
- you want a simple scriptable crawler without a web UI or job configuration
- you need JavaScript-heavy rendering out of the box without extra setup
Facets
application · maturity active
web-scraping http-client configuration-management crawlers web-development jvm web-crawler warc web-archiving digital-preservation internet-archive robots-txt archival-quality archives search linux docker web-server
1 source
- readme: https://github.com/internetarchive/heritrix3 · fetched 2026-08-28 · d99049684fac
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| internetarchive/heritrix3 | main | 95 |
For agents
markdown · JSON · MCP: product_card(name="internetarchive/heritrix3")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem