danielmiessler/RobotsDisallowed resource
A curated list of the most common and most interesting robots.txt disallowed directories. observed · 2026-08-28
Health v2 · maintenance only
32/100
- Activity 0
- Release rhythm 35
- Longevity 100
Flags: no_releases no_license
How is this computed?
round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10) — computed 2026-09-03. Adoption (stars, forks) is never an input.
- gap_med: n/a
- age_days: 3999
- days_rel: n/a
- days_push: 1472
- n_releases_24m: 0
Adoption not part of the score
1490 stars · 302 forks observed · 2026-08-28
What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-30, confidence not recorded
A curated dataset of robots.txt disallowed directories harvested from the top 100K websites (Alexa/Majestic), useful for content discovery during web security assessments and bug bounties. It includes a focused curated.txt list of high-value entries like admin, login, and backup directories.
Use cases
- find hidden directories on a website during a pentest
- wordlist for directory brute-forcing in bug bounty recon
- discover sensitive paths like admin panels and backups
- supplement content discovery tooling like gobuster or ffuf
- research common disallowed paths across top websites
When to choose
- you need a ready-made wordlist of interesting robots.txt paths for web assessments
- you want a small curated list of high-value targets instead of a huge generic wordlist
When to avoid
- you need an up-to-date dataset - the list was last refreshed in 2019
- you need a tool that fetches and parses robots.txt live rather than a static list
Facets
dataset · maturity maintenance
web-scraping security osint penetration-testing security penetration-testing crawlers osint cli cross-platform robots-txt wordlist bug-bounty content-discovery directory-enumeration curated-list
1 source
- readme: https://github.com/danielmiessler/RobotsDisallowed · fetched 2026-08-28 · ab51e81285a3
Member repositories
| Repository | Role | Health v2 |
|---|---|---|
| danielmiessler/RobotsDisallowed | main | 32 |
For agents
markdown · JSON · MCP: product_card(name="danielmiessler/RobotsDisallowed")
Data as of 2026-08-30T08:39:29.467469+00:00 · Report a problem