# watercrawl/WaterCrawl

Transform Web Content into LLM-Ready Data

Repository: https://github.com/watercrawl/WaterCrawl
Canonical: https://ross.abutalabs.com/products/watercrawl
Homepage: https://watercrawl.dev
Language: TypeScript
License: NOASSERTION
License Family: other
Topics: crawl4ai, crawler, crawling-python, html2markdown, llm-crawler, llm-scraper, scraper, aicrawler
Last push: 2026-08-17T11:44:41+00:00

## Health v2 (maintenance only)
Score: 82/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 98, release rhythm 84, longevity 44
- inputs: {"age_days": 626, "days_push": 16, "days_rel": 105, "gap_med": 9, "n_releases_24m": 28}
- flags: no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2010, forks 252 (observed 2026-08-28T04:06:05.064526+00:00)

## What it is
WaterCrawl is a self-hostable web application (Python/Django/Scrapy/Celery) that crawls websites and transforms web content into LLM-ready data like markdown, with sitemap generation, web search, JavaScript rendering, and AI-powered processing. It exposes a REST API with official Python client libraries and a hosted cloud offering with free and paid plans.

## Use cases
- convert website content into markdown for LLM training or RAG
- crawl a site and extract main content without ads and footers
- generate a sitemap of all URLs on a website
- scrape pages with JavaScript rendering and screenshots
- feed crawled web data into a chatbot or RAG pipeline
- run a self-hosted scraping API with proxy support

## When to choose
- you need LLM-ready structured data from websites
- you want a self-hosted alternative to hosted scraping APIs like Firecrawl
- you need sitemap discovery, search, and crawling in one service
- you want an API-first crawler with a Python client

## When to avoid
- you only need a lightweight one-off scraper script
- you require a permissive open-source license (license is not standard OSI)
- you need heavy client-side browser automation beyond rendering
- you cannot run Docker or don't want a multi-service stack (Django, Celery, MinIO, database)

## Facets
- artifact type: application
- maturity: active
- function: web-scraping, search-engine, rag, llm-inference, api-framework, self-hosted
- domain: crawlers, large-language-models, web-development, developer-tools
- platform: self-hosted, python, cli
- tags: crawler, scraper, html-to-markdown, django, scrapy, celery, sitemap-generation, javascript-rendering, api-service, data-extraction, retrieval-augmented-generation, docker, web-server

## Member repositories
- watercrawl/WaterCrawl (main) score 82

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:06:05.064526+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T03:01:15.059077+00:00, confidence not recorded.
  - readme: https://github.com/watercrawl/WaterCrawl (fetched 2026-08-28T04:06:05.064526+00:00, sha 47d1f81e03fa)
  - homepage: https://watercrawl.dev (fetched 2026-08-29T10:41:31.004895+00:00, sha 196daac95d9c)
  - site_page: https://docs.watercrawl.dev/clients/python (fetched 2026-08-29T10:41:31.046935+00:00, sha 9ffa21511a34)
  - site_page: https://docs.watercrawl.dev/ (fetched 2026-08-29T10:41:31.050806+00:00, sha 03566b4919e5)
  - site_page: https://docs.watercrawl.dev/api/documentation (fetched 2026-08-29T10:41:31.052416+00:00, sha 269d4448523f)
  - site_page: https://watercrawl.dev/changelog (fetched 2026-08-29T10:41:31.039714+00:00, sha 35b0c6db7d82)
  - site_page: https://watercrawl.dev/pricing (fetched 2026-08-29T10:41:31.049055+00:00, sha bb25210344a9)
- Data as of 2026-08-30T08:39:29.467469+00:00.