spider-rs/spider
Get web data for AI agents and LLMs - fast, efficient, and reliable via Rust observed · 2026-08-28
Health v2 · maintenance only
87/100
- Activity 98
- Release rhythm 65
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: 4.0
- age_days: 3160
- days_rel: 155
- days_push: 14
- n_releases_24m: 69
Adoption not part of the score
2672 stars · 281 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
Spider is a concurrency-first web crawler and scraper written in Rust that streams pages as they arrive, renders JavaScript only when needed, and outputs markdown, HTML, JSON, or text for AI agents and RAG pipelines. It runs fully locally as an open-source crate and also powers a managed cloud API with proxies, anti-bot bypass, and AI extraction.
Use cases
- crawl entire websites into markdown for LLM and RAG ingestion
- scrape single pages as structured JSON for AI agents
- build a search tool that fetches and scrapes live web results
- bypass bot protection with stealth mode and rotating proxies
- stream crawl results into S3, GCS, or Supabase data pipelines
- feed live web data into LangChain or LlamaIndex document loaders
- take full-page screenshots of rendered pages
- extract structured data from pages using AI prompts or CSS selectors
When to choose
- you need a fast, high-throughput crawler in Rust with fine-grained control
- you want clean markdown/JSON output for LLMs and RAG pipelines
- you need JavaScript rendering and anti-bot bypass at scale
- you want to prototype locally and scale to a managed service with one config change
- you need integrations with LangChain, LlamaIndex, CrewAI, or MCP clients
When to avoid
- you only need a simple one-off HTTP fetch without crawling features
- you require a language other than Rust for the core engine (SDKs exist but the engine is Rust)
- you need a fully free managed service - the cloud API is paid
- you need deep browser automation beyond scraping (use Playwright or Puppeteer)
Facets
library · maturity active
web-scraping http-client search-engine rag agent-framework crawlers web-development artificial-intelligence large-language-models developer-tools rust cross-platform cli web-crawler scraper headless-chrome javascript-rendering anti-bot-bypass markdown-output concurrent-crawling ai-data-pipeline mcp-server proxy-rotation retrieval-augmented-generation ai-agents web-server
10 sources
- readme: https://github.com/spider-rs/spider · fetched 2026-08-28 · b44a9083ca64
- homepage: https://spider.cloud/github · fetched 2026-08-29 · a14c27eb02ae
- site_page: https://spider.cloud/docs/overview · fetched 2026-08-29 · 0e48292f3758
- site_page: https://spider.cloud/about · fetched 2026-08-29 · 91e314e50ae3
- site_page: https://spider.cloud/docs/api · fetched 2026-08-29 · ed6c743a8e23
- site_page: https://spider.cloud/pricing · fetched 2026-08-29 · 2ef66b172b24
- site_page: https://spider.cloud/compare · fetched 2026-08-29 · 2b561a8da2c1
- site_page: https://spider.cloud/faq · fetched 2026-08-29 · 1887b4f46f3d
- site_page: https://spider.cloud/integrations · fetched 2026-08-29 · e2f5f8e56ea9
- site_page: https://spider.cloud/changelog · fetched 2026-08-29 · 3aac5a59fb15
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| spider-rs/spider | main | 87 |
For agents
markdown · JSON · MCP: product_card(name="spider-rs/spider")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem