rmax/scrapy-redis

Health v2 · maintenance only

Adoption not part of the score

5642 stars · 1583 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Redis-based components for Scrapy that enable distributed crawling and scraping by sharing a Redis queue across multiple spider instances. It provides plug-and-play Scrapy components including a scheduler with duplication filter, item pipeline, and base spiders.

Use cases

• run distributed crawls across multiple spider instances sharing one queue
• scale broad multi-domain web scraping with multiple workers
• push scraped items to a Redis queue for distributed post-processing
• deduplicate URLs across distributed Scrapy spiders
• pass JSON metadata with URLs through a Redis-backed request queue

When to choose

• you already use Scrapy and need to distribute crawling across machines
• you want plug-and-play Redis-backed scheduler and pipeline components
• you need multiple workers sharing a single crawl queue

When to avoid

• you need advanced features like URL expiration or prioritization (consider Frontera)
• you run a single-machine crawl with no need for distribution
• you don't use Scrapy as your crawling framework

Facets

library · maturity active

web-scraping message-queue caching middleware crawlers web-development databases developer-tools python cross-platform scrapy redis distributed-crawling spider crawler

Member repositories

For agents

markdown · JSON · MCP: product_card(name="rmax/scrapy-redis")