google/robotstxt

Health v2 · maintenance only

Adoption not part of the score

3471 stars · 253 forks observed · 2026-08-28

What it is AI-extracted, prompt v1, taxonomy v1, 2026-08-29, confidence not recorded

Google's production robots.txt parser and matcher, released as a C++ library (C++14 compliant). It implements the Robots Exclusion Protocol exactly as Googlebot uses it, plus a small CLI binary for testing URLs against a robots.txt file.

Use cases

• parse robots.txt files in a crawler
• check if a URL is allowed for a user-agent
• match robots.txt rules like Googlebot does
• build a web crawler that respects robots.txt
• test a robots.txt file against URLs
• implement the robots exclusion protocol

When to choose

• you need Google-identical robots.txt parsing semantics
• you're building a crawler or SEO tool in C++ or with C++ bindings
• you want the reference implementation of the Robots Exclusion Protocol

When to avoid

• you need robots.txt parsing in another language without C++ interop
• you want a high-level crawler framework rather than a parser library
• you need to generate robots.txt files rather than parse them

Facets

library · maturity stable

parser developer-tools web-development crawlers developer-tools windows cpp cross-platform robots-txt robots-exclusion-protocol web-crawling googlebot c-plus-plus-library linux macos

Member repositories

For agents

markdown · JSON · MCP: product_card(name="google/robotstxt")