gnemoug/distribute_crawler
使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现 observed · 2026-08-28
Health v2 · maintenance only
32/100
- Activity 0
- Release rhythm 35
- Longevity 100
Flags: no_releases no_license
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-02. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 4886
- days_rel: n/a
- days_push: 3424
- n_releases_24m: 0
Adoption not part of the score
3238 stars · 1560 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
A distributed web crawler built on Scrapy, Redis, MongoDB, and Graphite, demonstrated with a spider for a Chinese book-download site. Redis coordinates distributed request queues and stats, MongoDB (single or sharded cluster) stores scraped data and files via GridFS, and Graphite visualizes crawler status in real time.
Use cases
- build a distributed web crawler with scrapy and redis
- scrape a website and store results in mongodb
- monitor crawler stats in real time with graphite
- avoid getting banned while crawling by rotating user agents
- set up a local mongodb sharded cluster for scraping projects
- download images and files during web scraping with deduplication
When to choose
- you want a reference implementation of distributed crawling with scrapy-redis
- you need reusable scrapy pipelines and middlewares for mongodb storage and user-agent rotation
- you are exploring vertical search engine crawler architecture
When to avoid
- you need a maintained, up-to-date crawler framework - the project has been inactive since 2017 and has no license
- you want a simple one-off scraper without redis/mongodb/graphite infrastructure
- you need production support or documentation in English
Facets
application · maturity abandoned
web-scraping monitoring caching crawlers web-development python scrapy redis mongodb graphite distributed-crawler vertical-search-engine crawlers data-engineering search linux docker
1 source
- readme: https://github.com/gnemoug/distribute_crawler · fetched 2026-08-28 · 156d1c9123b6
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| gnemoug/distribute_crawler | main | 32 |
For agents
markdown · JSON · MCP: product_card(name="gnemoug/distribute_crawler")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem