# SpiderClub/weibospider

:zap: A distributed crawler for weibo, building with celery and requests.

Repository: https://github.com/SpiderClub/weibospider
Canonical: https://ross.abutalabs.com/products/spiderclub-weibospider
Language: Python
License: MIT
License Family: permissive
Topics: weibospider, data-analysis, python3, distributed-crawler, weibo, sina
Last push: 2020-07-11T04:35:54+00:00

## Health v2 (maintenance only)
Score: 32/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 0, release rhythm 35, longevity 100
- inputs: {"age_days": 3719, "days_push": 2244, "days_rel": null, "gap_med": null, "n_releases_24m": 0}
- flags: no_releases
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 4793, forks 1181 (observed 2026-08-28T04:08:59.952436+00:00)

## What it is
A distributed web crawler for Sina Weibo (Chinese microblogging platform) built with Python, Celery, and requests. It scrapes user profiles, keyword search results, posts, comments, and repost relationships, storing data in MySQL with an optional Django-based admin UI for configuration.

## Use cases
- scrape weibo user profiles and posts
- collect weibo search results for keywords
- harvest comments and reposts from weibo
- build a weibo dataset for social media analysis
- run a distributed crawler for sina weibo data

## When to choose
- you need structured weibo data (users, posts, comments, reposts) at scale
- you want a battle-tested crawler with account management and error handling built in
- you prefer a self-hosted Python stack with Celery workers and MySQL storage

## When to avoid
- you need data from platforms other than Weibo
- you want an actively maintained project - the last release was in 2020 and Weibo's API/pages have likely changed
- you need a simple one-off scrape without setting up MySQL, Redis, and Celery infrastructure

## Facets
- artifact type: application
- maturity: maintenance
- function: web-scraping, etl, data-science
- domain: social-media, crawlers, data-science, big-data
- platform: python, self-hosted
- tags: weibo, distributed-crawler, celery, sina-weibo, scraping, django-admin, mysql, redis, crawlers, linux, docker

## Member repositories
- SpiderClub/weibospider (main) score 32

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:08:59.952436+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:18:40.901979+00:00, confidence not recorded.
  - readme: https://github.com/SpiderClub/weibospider (fetched 2026-08-28T04:08:59.952436+00:00, sha 8499b5685da1)
- Data as of 2026-08-30T08:39:29.467469+00:00.