Measured from 76 real robots.txt files
Does Googlebot follow different robots.txt rules than everyone else?
Yes, on some sites. Re-testing the census hosts on 2026-08-29 found 7 of 76 files where the googlebot verdict disagreed with the star group verdict. The disagreements ran to 167 probe paths across those 7 hosts. This page shows the method and the numbers.
- 76hosts with a usable robots.txt in the re-test
- 7hosts where the googlebot verdict differed from the star-only verdict
- 167probe paths where the two verdicts disagreed
- 7of 7 differing hosts wrote a googlebot group
01The short answer: yes, on 7 of 76 hosts
Googlebot does not read robots.txt with different rules. It reads the same RFC 9309 file. The difference is which group in that file applies to it.
When a file contains a group for a specific user agent, that group is the only one that applies to that crawler. The * group is not merged in and not used as a fallback. Google states this plainly: user agent specific groups and global groups are not combined.
That is why a checker with no user agent field can mislead you. It applies the star group and stops. It never shows you the googlebot verdict when a googlebot group exists.
02The method, in numbered steps
The re-test used the same parser semantics as the robots.txt checker on this site, RFC 9309 with Google's own parser behaviour. Here is exactly what ran.
- Each of the 83 hosts named in the census perSite list was fetched again at
/robots.txtover HTTPS on 2026-08-29. - 76 returned a usable robots.txt body. Hosts returning nothing or an HTML page were excluded.
- For every file, each allow or disallow pattern was turned into a probe path. The
$was removed and*was replaced by the letter x. - Each probe path was evaluated twice. Once with the googlebot groups selected, and once with only the
*groups selected. Both used pattern-length priority, with strict-inequality ties going to allow. - A difference is one probe path where the two verdicts disagree.
The raw result is published beside this page.
03The 7 hosts that disagreed
Every host with a difference also had a googlebot group. That group is the only reason the verdicts can diverge. The probe path count comes from each file's own rules, so it differs per site.
| Host | Verdict differences | Probe paths | Has googlebot group |
|---|---|---|---|
| nytimes.com | 88 | 113 | yes |
| yelp.com | 58 | 253 | yes |
| indeed.com | 16 | 231 | yes |
| wsj.com | 2 | 51 | yes |
| imdb.com | 1 | 21 | yes |
| jstor.org | 1 | 25 | yes |
| reuters.com | 1 | 28 | yes |
| Totals | 167 | 722 | 7 of 7 |
Totals recompute from the census re-check file. If that file disagrees with this table, the file wins.
04What it means for SEOs
Your robots.txt tester must test the googlebot group separately from the star group. A tool with no user agent field cannot show you a difference, so it reports the star verdict as if it applied to Googlebot.
That can hide a real outcome. If a site keeps most restrictions in a * group and writes a thin googlebot group, Googlebot can end up allowed on paths the star group blocks. The parser confirms it: once a specific group matches, the global groups are not read at all.
The checker on this site has a user agent field and names the rule that decided. Paste a file, pick Googlebot, and test a path. Use the robots txt checker to see the googlebot verdict, not just the star verdict.
Frequently asked
Questions about Googlebot and robots rules
Does Googlebot use different robots.txt rules than other crawlers?
It can. Googlebot follows the group written for it when one exists. That group replaces the star group instead of being combined with it. On the 76 hosts re-tested on 2026-08-29, 7 files produced a googlebot verdict that disagreed with the star-only verdict, on 167 probe paths in total.
How many sites have a separate Googlebot group?
All 7 hosts with differing verdicts in the re-test had a googlebot group: nytimes.com, yelp.com, indeed.com, wsj.com, imdb.com, jstor.org, and reuters.com. A googlebot group is what lets the two verdicts diverge in the first place.
Does a robots.txt tester without a user agent field get it wrong?
It is incomplete, not always wrong. A checker with no user agent field can only apply the star group, so it cannot show the googlebot verdict. To see the difference you must test the googlebot group separately, which the robots.txt checker on this site does.
What does a verdict difference mean for a site's intent?
Nothing by itself. A difference says the googlebot verdict and the star-only verdict disagree for that path. It does not say which verdict the site intended. The re-test records the disagreement, not the intent.