# scrapy/scrapy

Scrapy, a fast high-level web crawling & scraping framework for Python.

Repository: https://github.com/scrapy/scrapy
Canonical: https://ross.abutalabs.com/products/scrapy
Homepage: https://scrapy.org
Language: Python
License: BSD-3-Clause
License Family: permissive
Topics: python, scraping, crawling, framework, crawler, hacktoberfest, web-scraping, web-scraping-python
Last push: 2026-08-26T16:38:28+00:00

## Health v2 (maintenance only)
Score: 99/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 98, longevity 100
- inputs: {"age_days": 6037, "days_push": 7, "days_rel": 13, "gap_med": 27.5, "n_releases_24m": 15}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 64048, forks 11919 (observed 2026-08-28T04:12:19.195542+00:00)

## What it is
Scrapy is a fast, high-level web crawling and scraping framework for Python used to extract structured data from websites. It provides a full project structure with spiders, selectors, item pipelines, and feed exports, backed by an async engine for large-scale crawling.

## Use cases
- scrape product prices from e-commerce sites
- crawl a website and extract structured data
- extract quotes and articles from web pages
- build a spider that recursively follows links
- export scraped data to JSON, CSV, or S3
- monitor web pages for content changes
- collect data for machine learning datasets

## When to choose
- you need to crawl many pages at scale with polite throttling
- you want a full project structure with pipelines and middlewares rather than a one-off script
- you need fast XPath/CSS selectors built in
- you want feed exports to multiple formats and storages
- you need an extensible framework with a large ecosystem of add-ons

## When to avoid
- you only need to parse a single HTML string without crawling
- you need to render heavy JavaScript pages without adding an extension like scrapy-playwright
- you want a lightweight requests-plus-BeautifulSoup script for a few pages
- your project is not Python-based

## Facets
- artifact type: framework
- maturity: stable
- function: web-scraping, http-client, parser, etl
- domain: crawlers, web-development
- platform: python, cross-platform, cli
- tags: crawler, spiders, xpath, css-selectors, scraping-framework, data-extraction, asyncio, twisted, data-engineering, automation

## Member repositories
- scrapy/scrapy (main) score 99

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:12:19.195542+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T16:17:25.879266+00:00, confidence not recorded.
  - readme: https://github.com/scrapy/scrapy (fetched 2026-08-28T04:12:19.195542+00:00, sha 5147a065b179)
  - homepage: https://scrapy.org (fetched 2026-08-28T17:55:48.627955+00:00, sha 25505e11b151)
  - site_page: https://docs.scrapy.org/en/latest/intro/tutorial.html (fetched 2026-08-28T17:55:48.636918+00:00, sha 8369bf52501a)
  - site_page: https://docs.scrapy.org/en/latest/topics/selectors.html (fetched 2026-08-28T17:55:48.639524+00:00, sha 770ad628a7e4)
  - site_page: https://docs.scrapy.org/en/latest/news.html (fetched 2026-08-28T17:55:48.642397+00:00, sha 0864e68def38)
  - site_page: https://docs.scrapy.org/en/latest/ (fetched 2026-08-28T17:55:48.646540+00:00, sha 507561d07f9c)
  - site_page: https://docs.scrapy.org/en/latest/faq.html (fetched 2026-08-28T17:55:48.648116+00:00, sha ffe82423216f)
  - registry_pypi: https://pypi.org/pypi/scrapy/json (fetched 2026-08-28T17:55:48.649932+00:00, sha edaacf64e424)
- Data as of 2026-08-30T08:39:29.467469+00:00.