Crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation. observed · 2026-08-28
Health v2 · maintenance only
98/100
- Activity 99
- Release rhythm 97
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: 17.0
- age_days: 3659
- days_rel: 21
- days_push: 7
- n_releases_24m: 27
Adoption not part of the score
25516 stars · 1643 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
Crawlee is a web scraping and browser automation library for Node.js/TypeScript (with a Python port) for building reliable crawlers. It integrates with Puppeteer, Playwright, Cheerio, and JSDOM, handling proxies, sessions, retries, and anti-blocking out of the box.
Use cases
- scrape product prices from e-commerce sites
- crawl a website and extract all links
- download HTML, PDF, and images from web pages
- collect training data for LLMs and RAG pipelines
- scrape pages that require JavaScript rendering
- avoid getting blocked while scraping at scale
- export scraped data to CSV or JSON
When to choose
- you need a full-featured scraping framework with proxy rotation and session management
- you want to scrape JavaScript-heavy sites with headless browsers
- you need reliable, scalable crawlers with retries and queueing
- you want TypeScript support and active maintenance
When to avoid
- you only need a one-off simple HTTP fetch without crawling logic
- you work in a language other than JavaScript/TypeScript or Python
- you need a distributed scraping service rather than an embedded library
Facets
library · maturity stable
web-scraping http-client browser-extension proxy file-upload crawlers web-development developer-tools artificial-intelligence cross-platform puppeteer playwright cheerio headless-browser proxy-rotation typescript crawler data-extraction llm-data automation nodejs docker
5 sources
- readme: https://github.com/apify/crawlee · fetched 2026-08-28 · ca1321af15c2
- homepage: https://crawlee.dev · fetched 2026-08-29 · fc7acc1ea741
- site_page: https://crawlee.dev/js/docs/guides · fetched 2026-08-29 · b58b16270a4f
- site_page: https://crawlee.dev/js/docs/examples · fetched 2026-08-29 · 30b7067a80a4
- site_page: https://crawlee.dev/js/api/core/changelog · fetched 2026-08-29 · 4319e57ca388
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| apify/crawlee | main | 98 |
| apify/crawlee-python | sdk | 89 |
For agents
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem