any4ai/AnyCrawl

Health v2 · maintenance only

Adoption not part of the score

3415 stars · 363 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

AnyCrawl is a Node.js/TypeScript web crawler and scraping service that converts websites into LLM-ready markdown/JSON data and extracts structured SERP results from search engines like Google, Bing, and Baidu. It offers a self-hostable OpenAPI-compatible HTTP API with multi-threaded bulk processing, Playwright-based rendering for JavaScript-heavy sites, scheduled tasks, webhooks, and an MCP server.

Use cases

• convert websites into clean markdown for LLM training or RAG pipelines
• scrape Google, Bing, and Baidu search results as structured JSON
• bulk crawl large sites with multi-threaded processing
• extract content from JavaScript-heavy SPAs and dynamic pages
• self-host a scraping API compatible with OpenAPI tooling
• schedule recurring crawl jobs with webhook notifications
• feed fresh web data to AI agents via MCP server

When to choose

• you need LLM-ready structured data from websites at scale
• you want SERP extraction from multiple search engines in one tool
• you prefer self-hosting a scraping service via Docker with zero configuration
• you need to handle JavaScript-rendered or dynamic content
• you want scheduled tasks, webhooks, and batch processing built in

When to avoid

• you only need a lightweight in-process scraping library without an HTTP service
• you require a fully managed hosted scraping API with no infrastructure
• your use case is simple single-page fetching that a plain HTTP client covers
• you need non-Node.js runtime integration as the primary dependency

Facets

service · maturity active

web-scraping http-server rag search-engine mcp json markdown scheduling webhook caching artificial-intelligence large-language-models crawlers web-development developer-tools self-hosted self-hosted cli serp-extraction html-to-markdown playwright multi-threading openapi batch-processing proxy-support llm-ready-data scraping-api firecrawl-alternative retrieval-augmented-generation search data-engineering nodejs docker web-server

Member repositories

For agents

markdown · JSON · MCP: product_card(name="any4ai/AnyCrawl")