Firecrawl

Health v2 · maintenance only

Adoption not part of the score

172809 stars · 9538 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Firecrawl is an open-source web scraping and crawling API that turns websites into clean Markdown, structured JSON, screenshots, and other LLM-ready formats. It handles JavaScript rendering, proxies, and rate limits, and is available both self-hosted and as a hosted service with an MCP server for AI agents.

Use cases

• scrape websites into markdown for llm training
• crawl an entire site and extract structured data
• feed fresh web content into a rag pipeline
• extract data from js-heavy single page apps
• convert web-hosted pdfs and docx files to text
• give my ai agent the ability to search and read the web
• scrape pages that require clicking or scrolling before extraction

When to choose

• you need reliable large-scale scraping without managing proxies or headless browsers
• you want clean markdown or structured output for LLMs and agents
• you need an MCP-ready web tool for AI agents
• you want both a hosted API and a self-hostable option

When to avoid

• you only need simple static HTML fetching with no JS rendering
• you need a fully free solution at high volume without self-hosting
• your target sites require heavy authenticated session workflows beyond Firecrawl's actions

Facets

service · maturity active

web-scraping search-engine parser rag mcp sdk crawlers artificial-intelligence large-language-models developer-tools python self-hosted cloud html-to-markdown data-extraction llm-ready-output crawler hosted-api agentic-search ai-agents nodejs web-server

Member repositories

For agents

markdown · JSON · MCP: product_card(name="firecrawl/firecrawl")