# unclecode/crawl4ai

🚀🤖 Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper. Don't be shy, join here: https://discord.gg/jP8KfhDhyN

Repository: https://github.com/unclecode/crawl4ai
Canonical: https://ross.abutalabs.com/products/crawl4ai
Homepage: https://crawl4ai.com
Language: Python
License: Apache-2.0
License Family: permissive
Last push: 2026-08-25T07:39:29+00:00

## Health v2 (maintenance only)
Score: 89/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 99, release rhythm 93, longevity 60
- inputs: {"age_days": 846, "days_push": 8, "days_rel": 49, "gap_med": 19, "n_releases_24m": 20}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 79471, forks 8233 (observed 2026-08-28T04:12:21.775559+00:00)

## What it is
Crawl4AI is an open-source Python library that crawls websites with a headless browser and converts pages into clean, LLM-ready Markdown for RAG pipelines, agents, and data extraction. It offers async crawling, CSS/XPath and LLM-based extraction strategies, adaptive crawling, and an optional Docker API server.

## Use cases
- scrape websites into markdown for RAG
- crawl docs to feed an LLM agent
- extract structured data from web pages
- crawl dynamic JavaScript-heavy pages
- build a data pipeline from web content
- stop crawling once enough info is gathered

## When to choose
- you need LLM-friendly markdown output from websites
- you want an async Python crawler with headless browser support
- you need both traditional CSS/XPath and LLM-based extraction
- you want adaptive crawling that stops when information is sufficient

## When to avoid
- you need a simple HTTP fetch without JavaScript rendering
- you want a no-code scraping service
- you need massive-scale distributed crawling infrastructure

## Facets
- artifact type: library
- maturity: active
- function: web-scraping, rag, llm-inference, parser, etl
- domain: crawlers, artificial-intelligence, large-language-models, developer-tools
- platform: python, cross-platform, cli
- tags: crawler, scraper, markdown-conversion, headless-browser, playwright, llm-friendly, async, data-extraction, retrieval-augmented-generation, data-engineering, docker

## Member repositories
- unclecode/crawl4ai (main) score 89

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:12:21.775559+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T16:14:03.380942+00:00, confidence not recorded.
  - readme: https://github.com/unclecode/crawl4ai (fetched 2026-08-28T04:12:21.775559+00:00, sha 6ceded5ff656)
  - homepage: https://crawl4ai.com (fetched 2026-08-28T17:41:54.803944+00:00, sha 75a54bf5642f)
  - site_page: https://docs.crawl4ai.com/core/quickstart/ (fetched 2026-08-28T17:41:54.812396+00:00, sha 07ba5dbc9542)
  - site_page: https://docs.crawl4ai.com/core/installation/ (fetched 2026-08-28T17:41:54.814362+00:00, sha 4861b3df084e)
  - site_page: https://docs.crawl4ai.com/advanced/advanced-features/ (fetched 2026-08-28T17:41:54.816400+00:00, sha 21ab5d411bc7)
  - site_page: https://docs.crawl4ai.com/core/adaptive-crawling/ (fetched 2026-08-28T17:41:54.818153+00:00, sha 3a169d20161e)
  - registry_pypi: https://pypi.org/pypi/crawl4ai/json (fetched 2026-08-28T17:41:54.819919+00:00, sha b45079a751b3)
- Data as of 2026-08-30T08:39:29.467469+00:00.