spatie/crawler
Some data may be outdated — see provenance labels.
https://spatie.be/docs/crawler observed · 2026-09-29
Health v2 · maintenance only
91/100
- Activity 88
- Release rhythm 89
- Longevity 100
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-10-21. Adoption (stars, forks) is never an input.
- gap_med: 16.5
- age_days: 4005
- days_rel: 74
- days_push: 74
- n_releases_24m: 21
Adoption not part of the score
2831 stars · 367 forks observed · 2026-09-29
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
A PHP library by Spatie for crawling links on websites, built on Guzzle promises for concurrent requests. It can execute JavaScript via Chrome and Puppeteer to crawl JavaScript-rendered sites.
Use cases
- crawl all links on a website
- check for broken links on my site
- collect all internal URLs from a domain
- crawl JavaScript-rendered single-page apps
- scrape website pages concurrently in PHP
When to choose
- you need a programmable web crawler in a PHP project
- you want concurrent crawling with configurable depth and crawl profiles
- you need to crawl sites that render content with JavaScript
When to avoid
- you need a full-scale distributed scraping system
- your stack is not PHP
- you only need a one-off CLI site audit without writing code
Facets
library · maturity active
web-scraping http-client concurrency crawlers web-development developer-tools php cli crawler guzzle link-checking javascript-rendering puppeteer
3 sources
- readme: https://github.com/spatie/crawler · fetched 2026-09-29 · 43361ac669e9
- homepage: https://freek.dev/308-building-a-crawler-in-php · fetched 2026-08-29 · e89fc461fc36
- site_page: https://freek.dev/about · fetched 2026-08-29 · e0ef9dccbb05
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| spatie/crawler | main | 91 |
For agents
Data as of 2026-10-19T08:05:51.212177+00:00 · Report a problem