crawlab-team/crawlab

Health v2 · maintenance only

Adoption not part of the score

12262 stars · 1888 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Crawlab is a Go-based distributed web crawler management platform with a web UI for managing, scheduling, and monitoring spiders written in any language or framework (Scrapy, Puppeteer, Selenium, etc.). It supports multi-node clusters over gRPC, a built-in Monaco code editor, task logs, and automatic storage of scraped data into MongoDB or SQL databases via native ORM.

Use cases

• manage and schedule web crawlers from a central dashboard
• run distributed scraping across multiple worker nodes
• monitor spider task logs and results in real time
• store scraped data into MySQL, PostgreSQL, or MongoDB automatically
• edit and deploy spider scripts in the browser
• replace scrapyd with a multi-language crawler platform

When to choose

• you run many spiders in different languages or frameworks and need unified management
• you need distributed crawling with master/worker nodes and scheduling
• you want a self-hosted admin UI with logs, results, and database integration

When to avoid

• you only need a single simple scraper script without orchestration
• you want a scraping framework/library to write crawler code rather than a management platform
• you cannot run Docker or MongoDB in your environment

Facets

application · maturity active

web-scraping scheduling monitoring logging self-hosted gui orm database crawlers web-development self-hosted self-hosted cross-platform go web-crawler spider-management distributed-crawling scrapy scrapyd task-scheduling grpc monaco-editor data-extraction data-engineering automation docker web-server linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="crawlab-team/crawlab")