ai-robots-txt/ai.robots.txt resource
A list of AI agents and robots to block. observed · 2026-08-28
Health v2 · maintenance only
92/100
- Activity 99
- Release rhythm 99
- Longevity 63
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: 13
- age_days: 889
- days_rel: 7
- days_push: 7
- n_releases_24m: 36
Adoption not part of the score
4082 stars · 178 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded
A community-maintained list of AI crawler user agents, distributed as robots.txt plus ready-made blocking configs for Apache, Nginx, Caddy, HAProxy, and lighttpd. It lets website owners block AI training and scraping bots from their sites.
Use cases
- block AI crawlers from my website
- generate a robots.txt that disallows AI bots
- block GPTBot and ClaudeBot in nginx
- stop AI companies from scraping my content for training
- configure HAProxy to deny AI user agents
- find a list of AI scraper user agents
When to choose
- you self-host a website and want to block AI crawlers at the server or robots.txt level
- you want a regularly updated, community-maintained list of AI bot user agents
- you need drop-in config snippets for common web servers
When to avoid
- you need to block bots that ignore robots.txt and spoof user agents - use behavioral detection or a WAF instead
- you run a CDN or managed platform where you cannot edit server configs
- you want to allow some AI crawlers but block others with fine-grained policy
Facets
dataset · maturity active
security web-scraping configuration-management privacy web-development crawlers self-hosted self-hosted cli robots-txt ai-crawlers bot-blocking nginx apache caddy haproxy lighttpd user-agent-filtering web-server
1 source
- readme: https://github.com/ai-robots-txt/ai.robots.txt · fetched 2026-08-28 · 8a841826b8f8
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| ai-robots-txt/ai.robots.txt | main | 92 |
For agents
markdown · JSON · MCP: product_card(name="ai-robots-txt/ai.robots.txt")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem