Robots txt checker that shows which rule won and why

01Test a path against a robots file

Paste the robots.txt, name the crawler, and give the path you want to test. The path is everything after the host, starting with a slash, and the query string counts. Nothing is uploaded. The parser runs in your browser, and nothing you paste or type ever leaves it. The page's own traffic is a first-party count sent once when it opens and once if you click a checkout link, and neither request carries anything you entered.

Parsing happens in your browser. The one thing a browser cannot do is read another site's file, so the fetch button calls a small relay endpoint the author of this site runs on workers.dev. It requests exactly one path, /robots.txt, over https, follows at most five redirects and refuses private address space.

Nothing you type is stored, and the paste box works with no fetch at all.

02The rule that decides is the longest pattern, not the first line

Two things about precedence surprise people, and both change the answer.

The first is that order does not matter. RFC 9309 says the most specific match must be used, and the most specific match is the one with the most octets. A disallow on line two does not beat an allow on line ten. The longer pattern wins wherever it sits in the file.

The second is the tie. When an allow and a disallow score the same, the allow wins. Google's parser only treats a path as blocked when the disallow priority is strictly greater than the allow priority, so equal scores fall through to allowed.

priority   = length of the pattern string, wildcards included
blocked    = disallowPriority > allowPriority        # strict, ties go to allow
no pattern = priority 0, which counts as no rule at all

allow: /page      disallow: /*.ph      path /page.php5
       5 octets            5 octets    tie, so ALLOWED

That last example is Google's own, and it is where the docs' wording and the parser's arithmetic pull apart. The docs describe the winner as the rule that "matches more characters in the URL", which reads as counting characters against the URL, while the parser measures the pattern string itself, wildcards included. Measured that way, /page and /*.ph both score five and the tie goes to allow, which is also how Google's current spec resolves it, calling equal rules conflicting and applying the least restrictive one.

All six of Google's published examples resolve under the parser's reading, which is the reading the tool above uses. You do not need to check that yourself. Section 05 lists all six as test cases with their results.

03A group written for your crawler replaces the star group completely

This is the rule that catches people with real sites. If a robots.txt contains a group naming your crawler, that group is the only one that applies. The * group is not merged in, not inherited, and not used as a fallback for paths the specific group does not mention.

So a specific group holding a single unrelated rule silently switches off every restriction in the star group for that crawler. Google states it plainly, that user agent specific groups and global groups are not combined, and the parser backs it up by returning allowed without ever reading the global priorities once a specific group has matched.

It is not a theoretical problem. Re-fetching the census hosts on 2026-08-29 and testing each file's own rule patterns as the paths, googlebot's verdict differed from what the star group alone would say 167 times across 7 sites, among them nytimes.com, imdb.com, yelp.com, reuters.com and wsj.com. The re-test file is published beside this page. A checker without a user agent field cannot show you this, and most quick checkers skip that field.

04What 83 major sites actually put in their robots file

To test the parser against real input rather than invented cases, 93 well known hosts were fetched on 2026-08-24. 83 returned a body, and those files hold 20,703 rules across 1,023 groups. One of the 83, linkedin.com, answered with an HTML page rather than a robots file, so its snapshot contributes zero rules. The raw result is published beside this page.

Two numbers in that table are worth stopping on. 15 of the 83 sites still set a crawl delay, a directive Google has never supported and ignores completely, so those lines do nothing for Googlebot. And 39 sites now name at least one AI crawler in a user agent line, which was a rare thing to see two years ago.

Naming a crawler is not the same as blocking it. Some of these groups carry an allow rule rather than a disallow, and this count is a token match on the user agent line, nothing more. A site that lists two variants of one operator, PerplexityBot and Perplexity-User for instance, counts twice in this table, so a token's line count can run higher than the number of distinct sites naming it.

05The conformance suite this parser passes

A checker that cannot show its own test results is asking to be trusted rather than verified. These are the cases the matcher on this page is tested against, and all 39 pass. The six marked as Google's are the precedence examples published in Google's specification.

06Lines Google reads and lines Google throws away

Google's parser recognises exactly four fields. Everything else is discarded silently, which is why a robots.txt can look carefully written and still do nothing.

Two of those deserve a plain statement. crawl-delay has never been supported by Google, and the way to slow Googlebot down is to return a 429, 500 or 503, or to use the rate limit form in Search Console.

noindex in robots.txt stopped working on 1 September 2019, and a page you want out of the index needs a noindex meta tag on the page itself, which means the page must stay crawlable for Google to ever see it.

08Where these rules come from

By Michael Lip, who fetched and published the census on this page and wrote the conformance suite the parser passes.

Every behaviour in the matcher traces to one of four primary sources, and the census file is published so the figures above can be recomputed rather than believed.

• SPECRFC 9309, the Robots Exclusion Protocol, for the most specific match rule, the tie behaviour, group selection and the 500 KiB parsing floor.
• SPECGoogle's robots.txt specification, for the four supported fields, the wildcard table, the six precedence examples and the HTTP status handling.
• CODEgoogle/robotstxt, the production matcher, for the detail the prose gets wrong, that priority is the length of the pattern string and that ties fall through to allow.
• DATAThe census file, every figure in section 04, fetched 2026-08-24 and published unmodified.