# istresearch/scrapy-cluster

This Scrapy project uses Redis and Kafka to create a distributed on demand scraping cluster.

Repository: https://github.com/istresearch/scrapy-cluster
Canonical: https://ross.abutalabs.com/products/scrapy-cluster
Homepage: http://scrapy-cluster.readthedocs.io/
Language: Python
License: MIT
License Family: permissive
Topics: python, scrapy, kafka, redis, scraping, distributed
Archived: true
Last push: 2023-11-07T12:16:25+00:00

## Health v2 (maintenance only)
Score: 10/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 8, longevity 100
- inputs: {"age_days": 4159, "days_push": 1030, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: archived
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 1225, forks 320 (observed 2026-08-28T04:04:03.020865+00:00)

## What it is
Scrapy Cluster is a distributed web scraping framework built on Scrapy that uses Redis to coordinate crawl requests and Kafka as a data bus for submitting jobs and receiving results. It allows scaling many spider instances across machines with on-demand, dynamic crawling of arbitrary URLs.

## Use cases
- crawl millions of pages across a cluster of scrapy workers
- distribute seed urls to multiple spider instances via redis
- submit scraping jobs and consume results through kafka topics
- scale web crawlers horizontally without downtime
- coordinate throttling of crawls across many machines
- run on-demand scraping of arbitrary websites

## When to choose
- you need large-scale distributed crawling beyond a single scrapy instance
- you want kafka as an integration bus for crawl jobs and results
- you need dynamic, on-demand crawling of arbitrary urls
- you want to add or remove scraper nodes without losing data

## When to avoid
- you only need a simple single-machine scraper
- you don't want to operate redis, kafka, and zookeeper infrastructure
- you need actively maintained software with recent development
- your project requires python 2.7-era scrapy compatibility only

## Facets
- artifact type: framework
- maturity: maintenance
- function: web-scraping, message-queue, caching, streaming, etl
- domain: crawlers, big-data, microservices
- platform: python, self-hosted
- tags: scrapy, kafka, redis, distributed-crawling, spider-cluster, data-engineering, automation, docker, linux

## Member repositories
- istresearch/scrapy-cluster (main) score 10

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:04:03.020865+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T06:15:23.830595+00:00, confidence not recorded.
  - readme: https://github.com/istresearch/scrapy-cluster (fetched 2026-08-28T04:04:03.020865+00:00, sha fe2a76acc098)
- Data as of 2026-08-30T08:39:29.467469+00:00.