simplecrawler/simplecrawler
Flexible event driven crawler for node. observed · 2026-08-28
Health v2 · maintenance only
10/100
- Activity 0
- Release rhythm 8
- Longevity 100
Flags: archived
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-02. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 5376
- days_rel: n/a
- days_push: 2005
- n_releases_24m: 0
Adoption not part of the score
2134 stars · 350 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
simplecrawler is a flexible, event-driven web crawler library for Node.js with a configurable queue system, robots.txt support, and link discovery. It is explicitly unmaintained, so users are advised to migrate to alternatives.
Use cases
- crawl an entire website and archive its pages
- scrape all links from a site
- build a site search index by crawling pages
- download large websites to disk
- monitor website pages for changes
- extract resources from a domain respecting robots.txt
When to choose
- you need a low-level, highly configurable crawler base in Node.js
- you want event-driven control over fetching and link discovery
- you need a queue that can be frozen to disk and resumed
When to avoid
- you need a maintained library with active support
- you want a modern headless-browser-based crawler for JavaScript-rendered sites
- you are starting a new project (the README recommends migrating to alternatives)
Facets
library · maturity abandoned
web-scraping http-client caching crawlers web-development developer-tools web-crawler spider event-driven queue robots-txt unmaintained nodejs javascript
2 sources
- readme: https://github.com/simplecrawler/simplecrawler · fetched 2026-08-28 · dbefab6ab1f3
- registry_npm: https://registry.npmjs.org/simplecrawler · fetched 2026-08-29 · 0773218c45b7
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| simplecrawler/simplecrawler | main | 10 |
For agents
markdown · JSON · MCP: product_card(name="simplecrawler/simplecrawler")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem