istresearch/scrapy-cluster

Health v2 · maintenance only

Flags: archived

Adoption not part of the score

1225 stars · 320 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

Scrapy Cluster is a distributed web scraping framework built on Scrapy that uses Redis to coordinate crawl requests and Kafka as a data bus for submitting jobs and receiving results. It allows scaling many spider instances across machines with on-demand, dynamic crawling of arbitrary URLs.

Use cases

• crawl millions of pages across a cluster of scrapy workers
• distribute seed urls to multiple spider instances via redis
• submit scraping jobs and consume results through kafka topics
• scale web crawlers horizontally without downtime
• coordinate throttling of crawls across many machines
• run on-demand scraping of arbitrary websites

When to choose

• you need large-scale distributed crawling beyond a single scrapy instance
• you want kafka as an integration bus for crawl jobs and results
• you need dynamic, on-demand crawling of arbitrary urls
• you want to add or remove scraper nodes without losing data

When to avoid

• you only need a simple single-machine scraper
• you don't want to operate redis, kafka, and zookeeper infrastructure
• you need actively maintained software with recent development
• your project requires python 2.7-era scrapy compatibility only

Facets

framework · maturity maintenance

web-scraping message-queue caching streaming etl crawlers big-data microservices python self-hosted scrapy kafka redis distributed-crawling spider-cluster data-engineering automation docker linux

Member repositories

For agents

markdown · JSON · MCP: product_card(name="istresearch/scrapy-cluster")