# spider-rs/spider

Get web data for AI agents and LLMs - fast, efficient, and reliable via Rust

Repository: https://github.com/spider-rs/spider
Canonical: https://ross.abutalabs.com/products/spider
Homepage: https://spider.cloud/github
Language: Rust
License: MIT
License Family: permissive
Topics: crawler, rust, spider, headless-chrome, scraping, automation, ai-agent, web-crawler, web-scraping, web-data
Last push: 2026-08-19T13:42:57+00:00

## Health v2 (maintenance only)
Score: 87/100 (v2, computed 2026-09-03T02:39:23.370411+00:00)
- activity 98, release rhythm 65, longevity 100
- inputs: {"age_days": 3160, "days_push": 14, "days_rel": 155, "gap_med": 4.0, "n_releases_24m": 69}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 2672, forks 281 (observed 2026-08-28T04:07:09.528824+00:00)

## What it is
Spider is a concurrency-first web crawler and scraper written in Rust that streams pages as they arrive, renders JavaScript only when needed, and outputs markdown, HTML, JSON, or text for AI agents and RAG pipelines. It runs fully locally as an open-source crate and also powers a managed cloud API with proxies, anti-bot bypass, and AI extraction.

## Use cases
- crawl entire websites into markdown for LLM and RAG ingestion
- scrape single pages as structured JSON for AI agents
- build a search tool that fetches and scrapes live web results
- bypass bot protection with stealth mode and rotating proxies
- stream crawl results into S3, GCS, or Supabase data pipelines
- feed live web data into LangChain or LlamaIndex document loaders
- take full-page screenshots of rendered pages
- extract structured data from pages using AI prompts or CSS selectors

## When to choose
- you need a fast, high-throughput crawler in Rust with fine-grained control
- you want clean markdown/JSON output for LLMs and RAG pipelines
- you need JavaScript rendering and anti-bot bypass at scale
- you want to prototype locally and scale to a managed service with one config change
- you need integrations with LangChain, LlamaIndex, CrewAI, or MCP clients

## When to avoid
- you only need a simple one-off HTTP fetch without crawling features
- you require a language other than Rust for the core engine (SDKs exist but the engine is Rust)
- you need a fully free managed service - the cloud API is paid
- you need deep browser automation beyond scraping (use Playwright or Puppeteer)

## Facets
- artifact type: library
- maturity: active
- function: web-scraping, http-client, search-engine, rag, agent-framework
- domain: crawlers, web-development, artificial-intelligence, large-language-models, developer-tools
- platform: rust, cross-platform, cli
- tags: web-crawler, scraper, headless-chrome, javascript-rendering, anti-bot-bypass, markdown-output, concurrent-crawling, ai-data-pipeline, mcp-server, proxy-rotation, retrieval-augmented-generation, ai-agents, web-server

## Member repositories
- spider-rs/spider (main) score 87

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:07:09.528824+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T02:18:07.279987+00:00, confidence not recorded.
  - readme: https://github.com/spider-rs/spider (fetched 2026-08-28T04:07:09.528824+00:00, sha b44a9083ca64)
  - homepage: https://spider.cloud/github (fetched 2026-08-29T10:00:52.359556+00:00, sha a14c27eb02ae)
  - site_page: https://spider.cloud/docs/overview (fetched 2026-08-29T10:00:52.368974+00:00, sha 0e48292f3758)
  - site_page: https://spider.cloud/about (fetched 2026-08-29T10:00:52.375566+00:00, sha 91e314e50ae3)
  - site_page: https://spider.cloud/docs/api (fetched 2026-08-29T10:00:52.380709+00:00, sha ed6c743a8e23)
  - site_page: https://spider.cloud/pricing (fetched 2026-08-29T10:00:52.371620+00:00, sha 2ef66b172b24)
  - site_page: https://spider.cloud/compare (fetched 2026-08-29T10:00:52.373640+00:00, sha 2b561a8da2c1)
  - site_page: https://spider.cloud/faq (fetched 2026-08-29T10:00:52.377328+00:00, sha 1887b4f46f3d)
  - site_page: https://spider.cloud/integrations (fetched 2026-08-29T10:00:52.379034+00:00, sha e2f5f8e56ea9)
  - site_page: https://spider.cloud/changelog (fetched 2026-08-29T10:00:52.383601+00:00, sha 3aac5a59fb15)
- Data as of 2026-08-30T08:39:29.467469+00:00.