# gnemoug/distribute_crawler

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现

Repository: https://github.com/gnemoug/distribute_crawler
Canonical: https://ross.abutalabs.com/products/distribute_crawler
Language: Python
License Family: other
Last push: 2017-04-18T08:00:51+00:00

## Health v2 (maintenance only)
Score: 32/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 0, release rhythm 35, longevity 100
- inputs: {"age_days": 4886, "days_push": 3424, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases, no_license
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 3238, forks 1560 (observed 2026-08-28T04:07:50.433184+00:00)

## What it is
A distributed web crawler built on Scrapy, Redis, MongoDB, and Graphite, demonstrated with a spider for a Chinese book-download site. Redis coordinates distributed request queues and stats, MongoDB (single or sharded cluster) stores scraped data and files via GridFS, and Graphite visualizes crawler status in real time.

## Use cases
- build a distributed web crawler with scrapy and redis
- scrape a website and store results in mongodb
- monitor crawler stats in real time with graphite
- avoid getting banned while crawling by rotating user agents
- set up a local mongodb sharded cluster for scraping projects
- download images and files during web scraping with deduplication

## When to choose
- you want a reference implementation of distributed crawling with scrapy-redis
- you need reusable scrapy pipelines and middlewares for mongodb storage and user-agent rotation
- you are exploring vertical search engine crawler architecture

## When to avoid
- you need a maintained, up-to-date crawler framework - the project has been inactive since 2017 and has no license
- you want a simple one-off scraper without redis/mongodb/graphite infrastructure
- you need production support or documentation in English

## Facets
- artifact type: application
- maturity: abandoned
- function: web-scraping, monitoring, caching
- domain: crawlers, web-development
- platform: python
- tags: scrapy, redis, mongodb, graphite, distributed-crawler, vertical-search-engine, crawlers, data-engineering, search, linux, docker

## Member repositories
- gnemoug/distribute_crawler (main) score 32

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:07:50.433184+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-30T07:24:32.829916+00:00, confidence not recorded.
  - readme: https://github.com/gnemoug/distribute_crawler (fetched 2026-08-28T04:07:50.433184+00:00, sha 156d1c9123b6)
- Data as of 2026-08-30T08:39:29.467469+00:00.