unclecode/crawl4ai

Health v2 · maintenance only

Adoption not part of the score

79471 stars · 8233 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Crawl4AI is an open-source Python library that crawls websites with a headless browser and converts pages into clean, LLM-ready Markdown for RAG pipelines, agents, and data extraction. It offers async crawling, CSS/XPath and LLM-based extraction strategies, adaptive crawling, and an optional Docker API server.

Use cases

• scrape websites into markdown for RAG
• crawl docs to feed an LLM agent
• extract structured data from web pages
• crawl dynamic JavaScript-heavy pages
• build a data pipeline from web content
• stop crawling once enough info is gathered

When to choose

• you need LLM-friendly markdown output from websites
• you want an async Python crawler with headless browser support
• you need both traditional CSS/XPath and LLM-based extraction
• you want adaptive crawling that stops when information is sufficient

When to avoid

• you need a simple HTTP fetch without JavaScript rendering
• you want a no-code scraping service
• you need massive-scale distributed crawling infrastructure

Facets

library · maturity active

web-scraping rag llm-inference parser etl crawlers artificial-intelligence large-language-models developer-tools python cross-platform cli crawler scraper markdown-conversion headless-browser playwright llm-friendly async data-extraction retrieval-augmented-generation data-engineering docker

Member repositories

For agents

markdown · JSON · MCP: product_card(name="unclecode/crawl4ai")