# Crawlee

Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation.

Repository: https://github.com/apify/crawlee
Canonical: https://ross.abutalabs.com/products/crawlee
Homepage: https://crawlee.dev
Language: TypeScript
License: Apache-2.0
License Family: permissive
Topics: web-scraping, web-crawling, npm, headless-chrome, puppeteer, automation, apify, scraping, crawling, crawler, headless, scraper, web-crawler, javascript, nodejs, playwright, typescript
Last push: 2026-08-26T20:43:47+00:00
Link (homepage): https://crawlee.dev
Link (site_page): https://crawlee.dev/js/docs/guides
Link (site_page): https://crawlee.dev/js/docs/examples
Link (site_page): https://crawlee.dev/js/api/core/changelog

## Health v2 (maintenance only)
Score: 98/100 (v2, computed 2026-09-03T02:20:16.233290+00:00)
- activity 99, release rhythm 97, longevity 100
- inputs: {"age_days": 3659, "days_push": 7, "days_rel": 21, "gap_med": 17.0, "n_releases_24m": 27}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 25516, forks 1643 (observed 2026-08-28T04:11:38.215786+00:00)

## What it is
Crawlee is a web scraping and browser automation library for Node.js/TypeScript (with a Python port) for building reliable crawlers. It integrates with Puppeteer, Playwright, Cheerio, and JSDOM, handling proxies, sessions, retries, and anti-blocking out of the box.

## Use cases
- scrape product prices from e-commerce sites
- crawl a website and extract all links
- download HTML, PDF, and images from web pages
- collect training data for LLMs and RAG pipelines
- scrape pages that require JavaScript rendering
- avoid getting blocked while scraping at scale
- export scraped data to CSV or JSON

## When to choose
- you need a full-featured scraping framework with proxy rotation and session management
- you want to scrape JavaScript-heavy sites with headless browsers
- you need reliable, scalable crawlers with retries and queueing
- you want TypeScript support and active maintenance

## When to avoid
- you only need a one-off simple HTTP fetch without crawling logic
- you work in a language other than JavaScript/TypeScript or Python
- you need a distributed scraping service rather than an embedded library

## Facets
- artifact type: library
- maturity: stable
- function: web-scraping, http-client, browser-extension, proxy, file-upload
- domain: crawlers, web-development, developer-tools, artificial-intelligence
- platform: cross-platform
- tags: puppeteer, playwright, cheerio, headless-browser, proxy-rotation, typescript, crawler, data-extraction, llm-data, automation, nodejs, docker

## Member repositories
- apify/crawlee (main) score 98
- apify/crawlee-python (sdk) score 89

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:11:38.215786+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T16:56:02.543013+00:00, confidence not recorded.
  - readme: https://github.com/apify/crawlee (fetched 2026-08-28T04:11:38.215786+00:00, sha ca1321af15c2)
  - homepage: https://crawlee.dev (fetched 2026-08-29T07:53:16.889473+00:00, sha fc7acc1ea741)
  - site_page: https://crawlee.dev/js/docs/guides (fetched 2026-08-29T07:53:16.898552+00:00, sha b58b16270a4f)
  - site_page: https://crawlee.dev/js/docs/examples (fetched 2026-08-29T07:53:16.900368+00:00, sha 30b7067a80a4)
  - site_page: https://crawlee.dev/js/api/core/changelog (fetched 2026-08-29T07:53:16.902350+00:00, sha 4319e57ca388)
- Data as of 2026-08-30T08:39:29.467469+00:00.