# 0xMassi/webclaw

Fast, local-first web content extraction for LLMs. Scrape, crawl, extract structured data — all from Rust. CLI, REST API, and MCP server.

Repository: https://github.com/0xMassi/webclaw
Canonical: https://ross.abutalabs.com/products/webclaw
Homepage: https://webclaw.io
Language: Rust
License: AGPL-3.0
License Family: copyleft
Topics: ai-agents, cli, llm, markdown, mcp, rust, self-hosted, web-extraction, web-scraping, web-crawler, ai-scraping, html-to-markdown, mcp-server, tls-fingerprinting, firecrawl-alternative, apify-alternative, scraperapi-alternative, scrapingbee-alternative, crawl4ai-alternative, jina-alternative
Last push: 2026-08-26T08:52:45+00:00

## Health v2 (maintenance only)
Score: 77/100 (v2, computed 2026-09-03T02:39:23.370411+00:00)
- activity 99, release rhythm 86, longevity 12
- inputs: {"age_days": 176, "days_push": 7, "days_rel": 17, "gap_med": 0.0, "n_releases_24m": 61}
- flags: young
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2305, forks 232 (observed 2026-08-28T04:06:35.672377+00:00)

## What it is
webclaw is a Rust-based web extraction toolkit that turns any URL into clean, LLM-ready markdown, JSON, or token-optimized text, including bot-protected pages. It ships as three binaries sharing one engine: a CLI, an axum-based REST API server, and an MCP server with 14 tools for AI clients like Claude and Cursor.

## Use cases
- scrape a webpage into clean markdown for an LLM prompt
- crawl an entire docs site and export every page as markdown
- convert HTML to LLM-optimized text with 90% fewer tokens
- extract structured JSON from a URL matching a schema
- plug live web access into Claude Desktop or Cursor via MCP
- monitor a webpage for changes with diffs
- self-host a Firecrawl alternative for RAG ingestion

## When to choose
- you need LLM-ready web content instead of raw HTML boilerplate
- you want a self-hostable, fast Rust alternative to Firecrawl or Jina
- you need MCP integration so AI agents can read the live web
- you scrape bot-protected sites and need TLS fingerprinting
- you want CLI, REST API, and MCP access from one extraction engine

## When to avoid
- you need a fully managed scraping service and don't want to self-host or pay for the cloud API
- you only need simple static HTML parsing in a language other than Rust
- your use case requires heavy JavaScript rendering at scale beyond what the engine handles
- you need a permissive license — the core is AGPL-3.0

## Facets
- artifact type: cli-tool
- maturity: active
- function: web-scraping, parser, rag, mcp, http-server, cli, search-engine
- domain: web-development, crawlers, large-language-models, developer-tools
- platform: cli, self-hosted, cross-platform, rust
- tags: html-to-markdown, firecrawl-alternative, mcp-server, llm-ready, tls-fingerprinting, rest-api, web-crawler, bot-protection-bypass, ai-agents, retrieval-augmented-generation, natural-language-processing, web-server

## Member repositories
- 0xMassi/webclaw (main) score 77

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:06:35.672377+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T02:39:41.295729+00:00, confidence not recorded.
  - readme: https://github.com/0xMassi/webclaw (fetched 2026-08-28T04:06:35.672377+00:00, sha 9d8d7c7b67c1)
  - homepage: https://webclaw.io (fetched 2026-08-29T10:20:07.633690+00:00, sha e09be3f832ff)
  - site_page: https://webclaw.io/docs (fetched 2026-08-29T10:20:07.644065+00:00, sha 767286522a9c)
  - site_page: https://webclaw.io/features (fetched 2026-08-29T10:20:07.648114+00:00, sha 7351cded2cf6)
  - site_page: https://webclaw.io/docs/api/scrape (fetched 2026-08-29T10:20:07.653372+00:00, sha bd6f2c8e0dad)
  - site_page: https://webclaw.io/docs/api/crawl (fetched 2026-08-29T10:20:07.654820+00:00, sha f1b722e5d5f9)
  - site_page: https://webclaw.io/docs/api/search (fetched 2026-08-29T10:20:07.656218+00:00, sha ee67c220f196)
  - site_page: https://webclaw.io/pricing (fetched 2026-08-29T10:20:07.646141+00:00, sha b8641c565109)
  - site_page: https://webclaw.io/integrations (fetched 2026-08-29T10:20:07.650183+00:00, sha 1627f0ff9b7f)
  - site_page: https://webclaw.io/pricing?billing=yearly (fetched 2026-08-29T10:20:07.651929+00:00, sha b8641c565109)
- Data as of 2026-08-30T08:39:29.467469+00:00.