unclecode/crawl4ai
Some data may be outdated — see provenance labels.
Open-source web crawler and scraper for LLMs and AI agents: any website into clean, LLM-ready Markdown. Run it yourself, or use Crawl4AI Cloud with one key. observed · 2026-10-02
Health v2 · maintenance only
89/100
- Activity 96
- Release rhythm 96
- Longevity 63
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-10-21. Adoption (stars, forks) is never an input.
- gap_med: 20
- age_days: 894
- days_rel: 27
- days_push: 25
- n_releases_24m: 22
Adoption not part of the score
84624 stars · 8756 forks observed · 2026-10-02
What it is AI-extracted, prompt v1, taxonomy v1, 2026-09-21, confidence 0.97
Crawl4AI is an open‑source Python library that provides an asynchronous, headless‑browser crawler and scraper optimized for LLM‑ready markdown output. It supports adaptive crawling, proxy routing, PDF generation, and screenshot capture for robust web data extraction.
Use cases
- extract clean markdown from web pages for retrieval‑augmented generation
- crawl dynamic JavaScript sites and capture screenshots or PDFs
- feed web content into large language model pipelines or agents
- perform large‑scale web scraping with proxy rotation and SSL handling
- collect data for training or fine‑tuning language models
When to choose
- You need a Python‑based crawler that outputs LLM‑friendly markdown.
- Your workflow requires asynchronous crawling of JavaScript‑heavy sites.
- You want built‑in adaptive crawling to avoid over‑ or under‑crawling.
- You need integrated proxy, PDF, or screenshot capabilities.
When to avoid
- You require a non‑Python solution or a low‑level HTTP client only.
- You need a fully managed cloud crawling service with no local dependencies.
- Your project cannot install Playwright or headless Chromium binaries.
Facets
library · maturity active
web-scraping web-development data-science artificial-intelligence python windows async markdown llm-friendly headless-browser playwright adaptive-crawling proxy-support pdf-screenshot linux macos docker
10 sources
- readme: https://github.com/unclecode/crawl4ai · fetched 2026-10-02 · 4cd9648a58ad
- homepage: https://crawl4ai.com · fetched 2026-09-07 · 75a54bf5642f
- site_page: https://docs.crawl4ai.com/core/quickstart/ · fetched 2026-08-28 · 07ba5dbc9542
- site_page: https://docs.crawl4ai.com/core/installation/ · fetched 2026-08-28 · 4861b3df084e
- site_page: https://docs.crawl4ai.com/advanced/advanced-features/ · fetched 2026-08-28 · 21ab5d411bc7
- site_page: https://docs.crawl4ai.com/core/adaptive-crawling/ · fetched 2026-08-28 · 3a169d20161e
- site_page: https://docs.crawl4ai.com/core/quickstart · fetched 2026-09-07 · 0690dd37aafe
- site_page: https://docs.crawl4ai.com/core/installation · fetched 2026-09-07 · b7aa36acf6d2
- site_page: https://docs.crawl4ai.com/advanced/advanced-features · fetched 2026-09-07 · 43b86d8a70f3
- site_page: https://docs.crawl4ai.com/core/adaptive-crawling · fetched 2026-09-07 · 0526cea6a90e
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| unclecode/crawl4ai | main | 89 |
For agents
markdown · JSON · MCP: product_card(name="unclecode/crawl4ai")
Data as of 2026-10-19T08:05:51.212177+00:00 · Report a problem