# any4ai/AnyCrawl

AnyCrawl 🚀: A Node.js/TypeScript crawler that turns websites into LLM-ready data and extracts structured SERP results from Google/Bing/Baidu/etc. Native multi-threading for bulk processing.

Repository: https://github.com/any4ai/AnyCrawl
Canonical: https://ross.abutalabs.com/products/anycrawl
Homepage: https://anycrawl.dev
Language: TypeScript
License: MIT
License Family: permissive
Topics: aitools, crawl, scrape, serp, webscraper, ai-scraping, data, html-to-markdown, rag, scraping
Last push: 2026-08-11T18:32:25+00:00

## Health v2 (maintenance only)
Score: 85/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 97, release rhythm 96, longevity 37
- inputs: {"age_days": 520, "days_push": 22, "days_rel": 28, "gap_med": 11, "n_releases_24m": 18}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 3415, forks 363 (observed 2026-08-28T04:08:03.685893+00:00)

## What it is
AnyCrawl is a Node.js/TypeScript web crawler and scraping service that converts websites into LLM-ready markdown/JSON data and extracts structured SERP results from search engines like Google, Bing, and Baidu. It offers a self-hostable OpenAPI-compatible HTTP API with multi-threaded bulk processing, Playwright-based rendering for JavaScript-heavy sites, scheduled tasks, webhooks, and an MCP server.

## Use cases
- convert websites into clean markdown for LLM training or RAG pipelines
- scrape Google, Bing, and Baidu search results as structured JSON
- bulk crawl large sites with multi-threaded processing
- extract content from JavaScript-heavy SPAs and dynamic pages
- self-host a scraping API compatible with OpenAPI tooling
- schedule recurring crawl jobs with webhook notifications
- feed fresh web data to AI agents via MCP server

## When to choose
- you need LLM-ready structured data from websites at scale
- you want SERP extraction from multiple search engines in one tool
- you prefer self-hosting a scraping service via Docker with zero configuration
- you need to handle JavaScript-rendered or dynamic content
- you want scheduled tasks, webhooks, and batch processing built in

## When to avoid
- you only need a lightweight in-process scraping library without an HTTP service
- you require a fully managed hosted scraping API with no infrastructure
- your use case is simple single-page fetching that a plain HTTP client covers
- you need non-Node.js runtime integration as the primary dependency

## Facets
- artifact type: service
- maturity: active
- function: web-scraping, http-server, rag, search-engine, mcp, json, markdown, scheduling, webhook, caching
- domain: artificial-intelligence, large-language-models, crawlers, web-development, developer-tools, self-hosted
- platform: self-hosted, cli
- tags: serp-extraction, html-to-markdown, playwright, multi-threading, openapi, batch-processing, proxy-support, llm-ready-data, scraping-api, firecrawl-alternative, retrieval-augmented-generation, search, data-engineering, nodejs, docker, web-server

## Member repositories
- any4ai/AnyCrawl (main) score 85

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:08:03.685893+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:38:28.101408+00:00, confidence not recorded.
  - readme: https://github.com/any4ai/AnyCrawl (fetched 2026-08-28T04:08:03.685893+00:00, sha 4bb9b807e186)
  - homepage: https://anycrawl.dev (fetched 2026-08-29T09:32:25.444486+00:00, sha ca7d89c3a621)
  - site_page: https://docs.anycrawl.dev (fetched 2026-08-29T09:32:25.450324+00:00, sha 34bb2f1d12ca)
  - site_page: https://anycrawl.dev/price (fetched 2026-08-29T09:32:25.448320+00:00, sha 4b94807942c8)
- Data as of 2026-08-30T08:39:29.467469+00:00.