monperrus/crawler-user-agents resource

Health v2 · maintenance only

Flags: no_releases

Adoption not part of the score

1400 stars · 289 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded

A curated JSON dataset of regular-expression patterns matching HTTP user-agents used by bots, crawlers, spiders, and scrapers. It is distributed as packages for JavaScript/Node.js, Python, and Go, with helper functions like is_crawler/IsCrawler for bot detection.

Use cases

• detect bot traffic in web server logs
• block crawlers and scrapers from hitting my API
• check if a user-agent string is a search engine bot
• filter out spider requests from analytics
• identify known bots by user-agent header
• exclude crawler traffic from rate limiting

When to choose

• you need a simple, regex-based list of known bot user-agents
• you want bot detection in Node.js, Python, or Go without heavy dependencies
• you need to filter or flag automated traffic in logs or middleware

When to avoid

• you need behavioral bot detection beyond user-agent strings
• you need a continuously updated commercial bot-fingerprinting service
• you need to detect malicious bots that spoof legitimate user-agents

Facets

dataset · maturity active

data-generation security http-server middleware web-development security developer-tools crawlers python go cross-platform user-agents bot-detection regex-patterns crawler-detection -dataset nodejs

Member repositories

For agents

markdown · JSON · MCP: product_card(name="monperrus/crawler-user-agents")