bda-research/node-crawler

Health v2 · maintenance only

Adoption not part of the score

6799 stars · 864 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

node-crawler is a TypeScript web crawler/spider library for Node.js that fetches pages and provides server-side DOM parsing with automatic jQuery-style manipulation via Cheerio. It offers a configurable request pool, retries, rate limiting, priority queues, and charset detection.

Use cases

• scrape product prices from e-commerce sites
• crawl a website and extract links
• parse html pages with jquery-like selectors in node
• build a polite crawler with rate limiting and retries
• extract data from web pages into structured output
• queue large-scale crawling jobs with priority

When to choose

• you need server-side DOM traversal with Cheerio/jQuery syntax
• you want built-in rate limiting, retries, and a priority queue
• you're on Node.js 22+ and comfortable with ESM
• you need charset detection and conversion handled automatically

When to avoid

• you need to render JavaScript-heavy SPA pages (use Puppeteer/Playwright instead)
• you want a simple one-off HTTP fetch without crawling features
• you're on an older Node.js version below 22

Facets

library · maturity active

web-scraping parser http-client caching crawlers web-development developer-tools cli cheerio jquery-server-side rate-limiting priority-queue crawler spider esm typescript nodejs

Member repositories

For agents

markdown · JSON · MCP: product_card(name="bda-research/node-crawler")