# binux/pyspider

A Powerful Spider(Web Crawler) System in Python.

Repository: https://github.com/binux/pyspider
Canonical: https://ross.abutalabs.com/products/pyspider
Homepage: http://docs.pyspider.org/
Language: Python
License: Apache-2.0
License Family: permissive
Topics: python, crawler
Archived: true
Last push: 2024-04-30T19:43:29+00:00

## Health v2 (maintenance only)
Score: 10/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 0, release rhythm 8, longevity 100
- inputs: {"age_days": 4576, "days_push": 855, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: archived
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 16774, forks 3625 (observed 2026-08-28T04:11:17.229403+00:00)

## What it is
pyspider is a powerful web crawler (spider) system written in Python with a built-in WebUI for script editing, task monitoring, project management, and result viewing. It supports distributed architecture, JavaScript page rendering via PhantomJS, multiple database and message queue backends, and task scheduling with priorities, retries, and periodic recrawls.

## Use cases
- crawl and scrape websites with a Python script
- schedule periodic recrawls of pages by age
- scrape JavaScript-rendered pages
- monitor and manage multiple crawling projects via a web UI
- run distributed crawling tasks with RabbitMQ or Redis queues
- store scraped results in MySQL, MongoDB, Elasticsearch, or SQLite

## When to choose
- you want a full crawler system with a web-based script editor and task dashboard
- you need distributed crawling with pluggable message queues and databases
- you need scheduled, prioritized, retrying crawl tasks out of the box
- you need to render JavaScript pages during crawling

## When to avoid
- you need a lightweight, modern scraping library - consider Scrapy or requests+BeautifulSoup instead
- you need active community support or recent Python 3.10+ compatibility
- you want a visual point-and-click scraping interface
- you cannot host the WebUI on an internal network since it exposes command execution by default

## Facets
- artifact type: framework
- maturity: maintenance
- function: web-scraping, scheduling, message-queue, web-framework
- domain: crawlers, web-development
- platform: python, cross-platform, self-hosted
- tags: crawler, spider, scraping, webui, distributed-crawling, phantomjs, automation, data-engineering, web-server

## Member repositories
- binux/pyspider (main) score 10

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:11:17.229403+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T17:04:16.021229+00:00, confidence not recorded.
  - readme: https://github.com/binux/pyspider (fetched 2026-08-28T04:11:17.229403+00:00, sha 7ab4be1340fb)
  - homepage: http://docs.pyspider.org/ (fetched 2026-08-29T08:01:58.771261+00:00, sha 67e581079710)
  - site_page: http://docs.pyspider.org/en/latest/Quickstart (fetched 2026-08-29T08:01:58.368852+00:00, sha 31a7761552c4)
  - site_page: http://docs.pyspider.org/en/latest/About-Tasks (fetched 2026-08-29T08:01:58.525781+00:00, sha 51475ad7b39d)
  - site_page: http://docs.pyspider.org/en/latest/About-Projects (fetched 2026-08-29T08:01:58.564186+00:00, sha 72f25710d314)
  - site_page: http://docs.pyspider.org/en/latest/tutorial (fetched 2026-08-29T08:01:58.683151+00:00, sha 119f104eb7a2)
  - registry_pypi: https://pypi.org/pypi/pyspider/json (fetched 2026-08-29T08:01:58.865320+00:00, sha 32ebd9caeef9)
- Data as of 2026-08-30T08:39:29.467469+00:00.