scrapy/scrapy
Some data may be outdated — see provenance labels.
Scrapy, a fast high-level web crawling & scraping framework for Python. observed · 2026-09-28
Health v2 · maintenance only
97/100
- Activity 97
- Release rhythm 94
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-10-21. Adoption (stars, forks) is never an input.
- gap_med: 23
- age_days: 6085
- days_rel: 40
- days_push: 23
- n_releases_24m: 16
Adoption not part of the score
64503 stars · 11974 forks observed · 2026-09-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-09-21, confidence 0.97
Scrapy is a fast, high-level web crawling and scraping framework for Python. It provides tools for defining spiders, extracting data, and exporting it in various formats.
Use cases
- Extract product information from e‑commerce sites
- Collect news articles for sentiment analysis
- Gather public data for market research
- Monitor website changes and trigger alerts
- Scrape job listings for recruitment analytics
When to choose
- You need a full‑featured, extensible framework for large‑scale crawling
- You want built‑in support for asynchronous requests and polite throttling
- You prefer Python and need integration with other Python data‑science tools
When to avoid
- You only need simple one‑off HTML parsing (BeautifulSoup may be lighter)
- You require a GUI tool rather than code‑first approach
- Your project is constrained to environments without Python support
Facets
framework · maturity active
web-scraping http-client cli parser middleware data-science windows python async distributed extensible open-source web-scraping automation linux macos
9 sources
- readme: https://github.com/scrapy/scrapy · fetched 2026-09-28 · 5147a065b179
- homepage: https://scrapy.org · fetched 2026-09-07 · 25505e11b151
- site_page: https://docs.scrapy.org/en/latest/intro/tutorial.html · fetched 2026-09-07 · 8369bf52501a
- site_page: https://docs.scrapy.org/en/latest/topics/selectors.html · fetched 2026-09-07 · 770ad628a7e4
- site_page: https://docs.scrapy.org/en/latest/news.html · fetched 2026-09-07 · 0864e68def38
- site_page: https://docs.scrapy.org/en/latest/ · fetched 2026-08-28 · 507561d07f9c
- site_page: https://docs.scrapy.org/en/latest/faq.html · fetched 2026-09-07 · ffe82423216f
- site_page: https://docs.scrapy.org/en/latest · fetched 2026-09-07 · cc9b977904c3
- registry_pypi: https://pypi.org/pypi/scrapy/json · fetched 2026-09-07 · edaacf64e424
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| scrapy/scrapy | main | 97 |
For agents
Data as of 2026-10-19T08:05:51.212177+00:00 · Report a problem