# google/robotstxt

The repository contains Google's robots.txt parser and matcher as a C++ library (compliant to C++11).

Repository: https://github.com/google/robotstxt
Canonical: https://ross.abutalabs.com/products/robotstxt
Language: C++
License: Apache-2.0
License Family: permissive
Last push: 2026-04-01T12:46:49+00:00

## Health v2 (maintenance only)
Score: 67/100 (v2, computed 2026-09-02T17:46:02.011165+00:00)
- activity 75, release rhythm 39, longevity 100
- inputs: {"age_days": 2640, "days_push": 154, "days_rel": 194, "gap_med": null, "n_releases_24m": 1}
- flags: none
- formula: round(0.45*activity + 0.35*rhythm + 0.20*longevity); archived -> min(score, 10)

## Adoption (not part of the score)
Stars 3471, forks 253 (observed 2026-08-28T04:08:05.979252+00:00)

## What it is
Google's production robots.txt parser and matcher, released as a C++ library (C++14 compliant). It implements the Robots Exclusion Protocol exactly as Googlebot uses it, plus a small CLI binary for testing URLs against a robots.txt file.

## Use cases
- parse robots.txt files in a crawler
- check if a URL is allowed for a user-agent
- match robots.txt rules like Googlebot does
- build a web crawler that respects robots.txt
- test a robots.txt file against URLs
- implement the robots exclusion protocol

## When to choose
- you need Google-identical robots.txt parsing semantics
- you're building a crawler or SEO tool in C++ or with C++ bindings
- you want the reference implementation of the Robots Exclusion Protocol

## When to avoid
- you need robots.txt parsing in another language without C++ interop
- you want a high-level crawler framework rather than a parser library
- you need to generate robots.txt files rather than parse them

## Facets
- artifact type: library
- maturity: stable
- function: parser, developer-tools
- domain: web-development, crawlers, developer-tools
- platform: windows, cpp, cross-platform
- tags: robots-txt, robots-exclusion-protocol, web-crawling, googlebot, c-plus-plus-library, linux, macos

## Member repositories
- google/robotstxt (main) score 67

## Provenance
- Observed fields: from GitHub, fetched 2026-08-28T04:08:05.979252+00:00.
- Health v2: computed from the inputs above; adoption is never an input.
- Inferred fields (summary, facets, guidance): AI-extracted, prompt v1, taxonomy v1, on 2026-08-29T18:36:37.856848+00:00, confidence not recorded.
  - readme: https://github.com/google/robotstxt (fetched 2026-08-28T04:08:05.979252+00:00, sha 054c6f502a40)
- Data as of 2026-08-30T08:39:29.467469+00:00.