Blocked by robots.txt

Rule Severity and Score

Rule id

indexing.robots_txt_blocked

Group

Indexing

Severity of a failure

Error

Counts in the score

Yes, whenever it applies to the page

Required for a score

No

Replaces the check

robots

Condition and Exceptions

Condition

Probe robots: the final URL path is disallowed for Googlebot (or * when no Googlebot group) under RFC 9309 matching (longest match, Allow wins ties, * and $ wildcards) → fail error only when the site is declared as meant to be crawled (expect_crawl true, per audit or website); when the expectation is unknown (the default) or crawling is not expected → review, severity label warning, no penalty

Exceptions

No probe → not_checked; robots 4xx → pass (no restriction); robots 5xx → review; a file over 500 KiB is evaluated on its first 500 KiB; every 2xx answer is parsed, and a 2xx without content received → not_checked

Sources and Project Decision

Sources

RFC 9309: Robots Exclusion Protocol (IETF, Q80)
How Google Interprets the robots.txt Specification (Google for Developers, S14)

Project decision

Error only when the owner declared that the site must be crawlable (the "Should be crawled" setting of the audit or its website). With no declared expectation (the default, and always the case for the anonymous audit) the result is review with severity warning and never changes the score (plan 5.2). A file over 500 KiB is evaluated on its first 500 KiB, as Google does; every 2xx answer is parsed.