scrapy/scrapy
Scrapy, a fast high-level web crawling & scraping framework for Python. observed · 2026-08-28
Health v2 · maintenance only
99/100
- Activity 99
- Release rhythm 98
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: 27.5
- age_days: 6037
- days_rel: 13
- days_push: 7
- n_releases_24m: 15
Adoption not part of the score
64048 stars · 11919 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
Scrapy is a fast, high-level web crawling and scraping framework for Python used to extract structured data from websites. It provides a full project structure with spiders, selectors, item pipelines, and feed exports, backed by an async engine for large-scale crawling.
Use cases
- scrape product prices from e-commerce sites
- crawl a website and extract structured data
- extract quotes and articles from web pages
- build a spider that recursively follows links
- export scraped data to JSON, CSV, or S3
- monitor web pages for content changes
- collect data for machine learning datasets
When to choose
- you need to crawl many pages at scale with polite throttling
- you want a full project structure with pipelines and middlewares rather than a one-off script
- you need fast XPath/CSS selectors built in
- you want feed exports to multiple formats and storages
- you need an extensible framework with a large ecosystem of add-ons
When to avoid
- you only need to parse a single HTML string without crawling
- you need to render heavy JavaScript pages without adding an extension like scrapy-playwright
- you want a lightweight requests-plus-BeautifulSoup script for a few pages
- your project is not Python-based
Facets
framework · maturity stable
web-scraping http-client parser etl crawlers web-development python cross-platform cli crawler spiders xpath css-selectors scraping-framework data-extraction asyncio twisted data-engineering automation
8 sources
- readme: https://github.com/scrapy/scrapy · fetched 2026-08-28 · 5147a065b179
- homepage: https://scrapy.org · fetched 2026-08-28 · 25505e11b151
- site_page: https://docs.scrapy.org/en/latest/intro/tutorial.html · fetched 2026-08-28 · 8369bf52501a
- site_page: https://docs.scrapy.org/en/latest/topics/selectors.html · fetched 2026-08-28 · 770ad628a7e4
- site_page: https://docs.scrapy.org/en/latest/news.html · fetched 2026-08-28 · 0864e68def38
- site_page: https://docs.scrapy.org/en/latest/ · fetched 2026-08-28 · 507561d07f9c
- site_page: https://docs.scrapy.org/en/latest/faq.html · fetched 2026-08-28 · ffe82423216f
- registry_pypi: https://pypi.org/pypi/scrapy/json · fetched 2026-08-28 · edaacf64e424
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| scrapy/scrapy | main | 99 |
For agents
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem