binux/pyspider
A Powerful Spider(Web Crawler) System in Python. observed · 2026-08-28
Health v2 · maintenance only
10/100
- Activity 0
- Release rhythm 8
- Longevity 100
Flags: archived
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 4576
- days_rel: n/a
- days_push: 855
- n_releases_24m: 0
Adoption not part of the score
16774 stars · 3625 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
pyspider is a powerful web crawler (spider) system written in Python with a built-in WebUI for script editing, task monitoring, project management, and result viewing. It supports distributed architecture, JavaScript page rendering via PhantomJS, multiple database and message queue backends, and task scheduling with priorities, retries, and periodic recrawls.
Use cases
- crawl and scrape websites with a Python script
- schedule periodic recrawls of pages by age
- scrape JavaScript-rendered pages
- monitor and manage multiple crawling projects via a web UI
- run distributed crawling tasks with RabbitMQ or Redis queues
- store scraped results in MySQL, MongoDB, Elasticsearch, or SQLite
When to choose
- you want a full crawler system with a web-based script editor and task dashboard
- you need distributed crawling with pluggable message queues and databases
- you need scheduled, prioritized, retrying crawl tasks out of the box
- you need to render JavaScript pages during crawling
When to avoid
- you need a lightweight, modern scraping library - consider Scrapy or requests+BeautifulSoup instead
- you need active community support or recent Python 3.10+ compatibility
- you want a visual point-and-click scraping interface
- you cannot host the WebUI on an internal network since it exposes command execution by default
Facets
framework · maturity maintenance
web-scraping scheduling message-queue web-framework crawlers web-development python cross-platform self-hosted crawler spider scraping webui distributed-crawling phantomjs automation data-engineering web-server
7 sources
- readme: https://github.com/binux/pyspider · fetched 2026-08-28 · 7ab4be1340fb
- homepage: http://docs.pyspider.org/ · fetched 2026-08-29 · 67e581079710
- site_page: http://docs.pyspider.org/en/latest/Quickstart · fetched 2026-08-29 · 31a7761552c4
- site_page: http://docs.pyspider.org/en/latest/About-Tasks · fetched 2026-08-29 · 51475ad7b39d
- site_page: http://docs.pyspider.org/en/latest/About-Projects · fetched 2026-08-29 · 72f25710d314
- site_page: http://docs.pyspider.org/en/latest/tutorial · fetched 2026-08-29 · 119f104eb7a2
- registry_pypi: https://pypi.org/pypi/pyspider/json · fetched 2026-08-29 · 32ebd9caeef9
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| binux/pyspider | main | 10 |
For agents
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem