Crawlee
Crawlee—A web scraping and browser automation library for Node.js to build reliable crawlers. In JavaScript and TypeScript. Extract data for AI, LLMs, RAG, or GPTs. Download HTML, PDF, JPG, PNG, and other files from websites. Works with Puppeteer, Playwright, Cheerio, JSDOM, and raw HTTP. Both headful and headless mode. With proxy rotation. observed · 2026-10-07
Health v2 · maintenance only
98/100
- Activity 98
- Release rhythm 97
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-10-21. Adoption (stars, forks) is never an input.
- gap_med: 17.0
- age_days: 3707
- days_rel: 21
- days_push: 14
- n_releases_24m: 25
Adoption not part of the score
26010 stars · 1687 forks observed · 2026-10-07
What it is AI-extracted, prompt v1, taxonomy v1, 2026-10-12, confidence 0.97
Crawlee is an open‑source web scraping and browser automation library for JavaScript/TypeScript (and Python) that abstracts away headless browsers, HTTP clients, and proxy handling. It provides high‑level crawlers, data storage utilities, and a CLI to quickly scaffold reliable scrapers.
Use cases
- Extract product listings and prices from e‑commerce sites.
- Collect news articles and blog posts for a content aggregator.
- Download PDFs, images, or other files from a large website.
- Crawl sitemaps and index pages for SEO analysis.
- Gather data for training LLMs or building RAG pipelines.
- Automate form submissions and login flows for testing.
When to choose
- You need a Node.js‑centric solution that works with both headless browsers and plain HTTP.
- You want built‑in proxy rotation and anti‑bot evasion features.
- You prefer TypeScript typings and a CLI that scaffolds projects out of the box.
- Your project requires integration with Playwright, Puppeteer, Cheerio, or JSDOM.
When to avoid
- You are looking for a pure Python or Java library without a JavaScript runtime.
- Your scraping tasks are extremely lightweight and a full‑featured crawler adds unnecessary overhead.
- You need deep OS‑level browser control that only native Selenium bindings provide.
Facets
library · maturity active
http-client web-scraping workflow-automation web-development browser web-crawling headless-browser playwright puppeteer cheerio jsdom proxy-rotation typescript cli-tool proxy-management data-engineering nodejs docker
5 sources
- readme: https://github.com/apify/crawlee · fetched 2026-10-07 · a379a8278742
- homepage: https://crawlee.dev · fetched 2026-09-28 · fc7acc1ea741
- site_page: https://crawlee.dev/js/docs/guides · fetched 2026-09-28 · b58b16270a4f
- site_page: https://crawlee.dev/js/docs/examples · fetched 2026-09-28 · 30b7067a80a4
- site_page: https://crawlee.dev/js/api/core/changelog · fetched 2026-09-28 · 4319e57ca388
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| apify/crawlee | main | 98 |
| apify/crawlee-python | sdk | 93 |
For agents
Data as of 2026-10-19T08:05:51.212177+00:00 · Report a problem