# coder-hxl/x-crawl

Flexible Node.js AI-assisted crawler library

Repository: https://github.com/coder-hxl/x-crawl
Canonical: https://ross.abutalabs.com/products/x-crawl
Homepage: https://coder-hxl.github.io/x-crawl/
Language: TypeScript
License: MIT
License Family: permissive
Topics: crawl, crawler, nodejs, typescript, spider, flexible, puppeteer, javascript, multifunction, chromium, fingerprint, ai, ai-crawl
Last push: 2026-08-26T19:15:45+00:00

## Health v2 (maintenance only)
Score: 66/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 8, longevity 94
- inputs: {"age_days": 1319, "days_push": 7, "days_rel": 514, "gap_med": null, "n_releases_24m": 1}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1877, forks 116 (observed 2026-08-28T04:05:47.910917+00:00)

## What it is
x-crawl is a flexible Node.js crawler library that supports crawling dynamic pages, static pages, API data, and files, with optional AI assistance via Ollama and OpenAI integration. It includes features like device fingerprint spoofing, proxy rotation, retry logic, interval-based crawling, and priority queues.

## Use cases
- scrape product prices from e-commerce sites
- crawl dynamic pages rendered with JavaScript
- collect data from APIs and save files
- extract content from static HTML pages
- automate browser interactions like keyboard input and clicks
- avoid bot detection with device fingerprints and rotating proxies
- use AI to simplify scraping tasks

## When to choose
- you need a single Node.js library for both static and dynamic page crawling
- you want built-in retry, proxy rotation, and rate limiting without extra tooling
- you want optional AI assistance to reduce scraping boilerplate
- you need TypeScript support with strong typing

## When to avoid
- you need a distributed, large-scale scraping framework like Scrapy
- you only need simple HTTP fetching without browser automation
- you require a language other than JavaScript/TypeScript
- you need guaranteed anti-bot bypass for heavily protected sites

## Facets
- artifact type: library
- maturity: active
- function: web-scraping, http-client, workflow-automation
- domain: crawlers, web-development, artificial-intelligence, developer-tools
- platform: cross-platform
- tags: crawler, spider, puppeteer, ai-assisted, fingerprint-spoofing, proxy-rotation, typescript, nodejs

## Member repositories
- coder-hxl/x-crawl (main) score 66

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:05:47.910917+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:14:41.905864+00:00, confidence not recorded.
  - readme: https://github.com/coder-hxl/x-crawl (fetched 2026-08-28T04:05:47.910917+00:00, sha f0eacd9714fe)
  - homepage: https://coder-hxl.github.io/x-crawl/ (fetched 2026-08-29T10:53:39.795633+00:00, sha 7fd78dc3fe8a)
  - registry_npm: https://registry.npmjs.org/x-crawl (fetched 2026-08-29T10:53:39.805522+00:00, sha 6140119c0af1)
- Data as of 2026-08-30T08:39:29.467469+00:00.