ArchiveTeam/grab-site
The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns observed · 2026-08-28
Health v2 · maintenance only
43/100
- Activity 23
- Release rhythm 35
- Longevity 100
Flags: no_releases no_license
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-02. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 4227
- days_rel: n/a
- days_push: 467
- n_releases_24m: 0
Adoption not part of the score
1607 stars · 156 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
grab-site is a preconfigured web crawler for archiving websites, producing WARC files via a fork of wpull. It includes a dashboard for monitoring all crawls, dynamic ignore patterns, and duplicate page detection.
Use cases
- back up an entire website to WARC files
- crawl a site recursively for archiving
- monitor multiple crawls from a dashboard
- add ignore patterns to a running crawl to skip junk URLs
- archive large sites with millions of pages
When to choose
- you need WARC-standard web archives
- you want a preconfigured crawler with sensible defaults
- you need to adjust ignore rules mid-crawl
- you're doing web preservation or Archive Team-style archiving
When to avoid
- you need a lightweight programmable scraping library rather than a full crawler
- you need a Python version other than 3.7/3.8 with pinned dependencies
- you need fine-grained custom crawl logic or API integration
- you only need to fetch a few pages rather than crawl a whole site
Facets
cli-tool · maturity active
web-scraping cli monitoring analytics crawlers developer-tools self-hosted windows python cli web-archiving warc crawler archiveteam backup dashboard linux macos
1 source
- readme: https://github.com/ArchiveTeam/grab-site · fetched 2026-08-28 · e0b209d3f11c
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| ArchiveTeam/grab-site | main | 43 |
For agents
markdown · JSON · MCP: product_card(name="ArchiveTeam/grab-site")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem