# internetarchive/heritrix3

Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project.

Repository: https://github.com/internetarchive/heritrix3
Canonical: https://ross.abutalabs.com/products/heritrix3
Homepage: https://heritrix.readthedocs.io/
Language: Java
License: NOASSERTION
License Family: other
Topics: java, webcrawling, warc, heritrix
Last push: 2026-08-25T02:06:42+00:00

## Health v2 (maintenance only)
Score: 95/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 87, longevity 100
- inputs: {"age_days": 5430, "days_push": 9, "days_rel": 9, "gap_med": 41.5, "n_releases_24m": 17}
- flags: no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 3305, forks 792 (observed 2026-08-28T04:07:55.816075+00:00)

## What it is
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler written in Java. It crawls websites and stores content in WARC archives while respecting robots.txt and politeness policies.

## Use cases
- crawl entire websites for archival purposes
- build web archive collections in WARC format
- mirror large portions of the web at scale
- respectfully crawl sites honoring robots.txt and politeness settings
- run scheduled recurring crawls of web resources
- extend crawler behavior with custom Java modules

## When to choose
- you need archival-quality, standards-compliant web crawling with WARC output
- you want a mature, battle-tested crawler used by the Internet Archive
- you need fine-grained politeness and robots.txt compliance
- you want an extensible crawler with a REST API and bean-based configuration

## When to avoid
- you only need lightweight scraping of a few pages
- you want a simple scriptable crawler without a web UI or job configuration
- you need JavaScript-heavy rendering out of the box without extra setup

## Facets
- artifact type: application
- maturity: active
- function: web-scraping, http-client, configuration-management
- domain: crawlers, web-development
- platform: jvm
- tags: web-crawler, warc, web-archiving, digital-preservation, internet-archive, robots-txt, archival-quality, archives, search, linux, docker, web-server

## Member repositories
- internetarchive/heritrix3 (main) score 95

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:07:55.816075+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:42:39.290122+00:00, confidence not recorded.
  - readme: https://github.com/internetarchive/heritrix3 (fetched 2026-08-28T04:07:55.816075+00:00, sha d99049684fac)
- Data as of 2026-08-30T08:39:29.467469+00:00.