coder-hxl/x-crawl

Health v2 · maintenance only

Adoption not part of the score

1877 stars · 116 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

x-crawl is a flexible Node.js crawler library that supports crawling dynamic pages, static pages, API data, and files, with optional AI assistance via Ollama and OpenAI integration. It includes features like device fingerprint spoofing, proxy rotation, retry logic, interval-based crawling, and priority queues.

Use cases

• scrape product prices from e-commerce sites
• crawl dynamic pages rendered with JavaScript
• collect data from APIs and save files
• extract content from static HTML pages
• automate browser interactions like keyboard input and clicks
• avoid bot detection with device fingerprints and rotating proxies
• use AI to simplify scraping tasks

When to choose

• you need a single Node.js library for both static and dynamic page crawling
• you want built-in retry, proxy rotation, and rate limiting without extra tooling
• you want optional AI assistance to reduce scraping boilerplate
• you need TypeScript support with strong typing

When to avoid

• you need a distributed, large-scale scraping framework like Scrapy
• you only need simple HTTP fetching without browser automation
• you require a language other than JavaScript/TypeScript
• you need guaranteed anti-bot bypass for heavily protected sites

Facets

library · maturity active

web-scraping http-client workflow-automation crawlers web-development artificial-intelligence developer-tools cross-platform crawler spider puppeteer ai-assisted fingerprint-spoofing proxy-rotation typescript nodejs

Member repositories

For agents

markdown · JSON · MCP: product_card(name="coder-hxl/x-crawl")