xianhu/PSpider
简单易用的Python爬虫框架，QQ交流群：597510560 observed · 2026-08-28
Health v2 · maintenance only
32/100
- Activity 0
- Release rhythm 35
- Longevity 100
Flags: no_releases
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 3609
- days_rel: n/a
- days_push: 1546
- n_releases_24m: 0
Adoption not part of the score
1835 stars · 494 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
PSpider is a simple, easy-to-read web spider framework written in Python 3.8+. It uses a Fetcher/Parser/Saver pipeline with queues for multi-threaded crawling and optional proxy support.
Use cases
- crawl websites with a multi-threaded python spider
- scrape web pages and save results to files or databases
- build a crawler that rotates proxies
- learn how a spider framework works with minimal code
- filter URLs while crawling to avoid duplicates
When to choose
- you want a small, readable Python crawler framework you can extend yourself
- you need multi-threaded crawling with proxy support
- you prefer copying a lightweight spider module into your project over a heavy dependency
When to avoid
- you need distributed crawling or JavaScript rendering (listed as TODO, not implemented)
- you need an actively developed framework with frequent updates and large ecosystem
- you need async/scrapy-scale features like middlewares, pipelines, and item processing
Facets
framework · maturity maintenance
web-scraping http-client parser concurrency crawlers web-development python cross-platform web-spider multi-threading proxies crawler-framework fetcher-parser-saver automation
2 sources
- readme: https://github.com/xianhu/PSpider · fetched 2026-08-28 · 1ab7f0803a3a
- homepage: https://github.com/xianhu/PSpider · fetched 2026-08-29 · 97c55145bb24
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| xianhu/PSpider | main | 32 |
For agents
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem