How was the robots.txt census done?

A fixed list of 93 well known hosts was fetched over HTTPS at /robots.txt with a 15 second timeout. The 83 that returned a body were parsed into user-agent groups and rules using a matcher that implements RFC 9309 and the behaviours in Google's open source parser robots.cc. Lines that were not user-agent, allow, disallow, or sitemap were counted as unknown directives.

Why does the biggest robots.txt file matter to SEOs?

The largest file in the census was figma.com at 380,638 bytes with 8,823 rules. Google stops reading a robots.txt past 500 KiB. None of the 83 hosts in this census hit that limit, but big files still cost crawl budget and are harder to maintain correctly.

Is the census a sample of the whole web?

No. It is a fixed list of 93 well known hosts, not a random sample. It describes how large, well resourced sites write robots.txt and says nothing about the average site. It is also a single snapshot taken on 2026-08-24, and several of these files are edited weekly.

Census, 2026-08-24

What share of websites ship a valid robots.txt?

Q: What share of websites ship a valid robots.txt?

In this census every one of the 83 hosts that served a robots.txt body shipped a file that parsed cleanly under RFC 9309. None of the 83 exceeded Google's 500 KiB file limit. The only failures were the 10 of 93 hosts that returned no body at all, and they are excluded from every figure.

We fetched robots.txt from 93 major hosts and parsed every response against RFC 9309 and Google's own parser. Every file that came back parsed cleanly. The failures show up before parsing, in the hosts that serve no file at all.

83hosts served a robots.txt body and all 83 parsed under RFC 9309
10of 93 attempted hosts returned no body and are excluded from every figure
20,703total rules counted across the 83 files
0files over Google's 500 KiB parsing ceiling

01The short answer

Every site that actually serves a robots.txt ships a valid one. All 83 hosts with a body parsed cleanly. None went over Google's 500 KiB ceiling.

That sounds like good news. It hides a real problem. The gap is at the start, not the end. Ten of the 93 hosts we tried returned no body at all. They answered with an HTTP error or refused the connection. A site with no reachable robots.txt gives a crawler nothing to respect.

Within the files that exist, the risks are different. They are not syntax failures. They are files so big and so cluttered that they are hard to trust. Figma ships 8,823 rules. Etsy ships 1,801. Those files parse, and then a human has to ask which rules actually do what they think they do.

02How the census was run

Every figure on this page recomputes from one census file. The method is four steps.

A fixed list of 93 well known hosts was fetched over HTTPS at /robots.txt with a 15 second timeout.
83 returned a response body. The other 10 returned an HTTP error or refused the connection, and they are excluded from every figure.
Each file was parsed into user-agent groups and allow or disallow rules. Lines that are not user-agent, allow, disallow, or sitemap are counted as unknown directives, which is what Google's parser does with them.
A sitemap line does not terminate a group, matching RFC 9309 section 2.2.4. The group counts are the ones a crawler would actually see.

The parser implements RFC 9309 and the behaviours in the google/robotstxt robots.cc source. Counting stops at the first 20,000 lines and 16,664 bytes per line, which mirrors the limits in Google's parser.

03The biggest files, by rule count

These eight hosts hold almost two thirds of all 20,703 rules. They are the files a crawler spends the most time on.

Census of 2026-08-24, top hosts by rule count
Host	Rules	Allow	Disallow	Groups	Bytes
figma.com	8,823	8,786	37	21	380,638
etsy.com	1,801	120	1,681	3	52,926
vimeo.com	972	614	358	15	22,642
ebay.com	965	77	888	9	24,363
airbnb.com	835	28	807	14	24,175
tripadvisor.com	718	12	706	7	25,115
indeed.com	505	37	468	9	13,097
wikipedia.org	464	4	460	33	28,275

Rule counts include every allow and disallow line, including rules inside groups that only apply to one named crawler. They are not the number of rules that apply to any single crawler.

04What this means for SEOs

Syntax is rarely the failure. The census files all parse. The failure modes are about size, reach, and honest maintenance.

SIZEBig files are a signal. Figma's 380,638-byte file and Etsy's 52,926-byte file are the outliers. Google stops reading at 500 KiB. You are well under that, but a giant file still costs crawl budget on every fetch and is far harder to keep correct.
SITEMAP62 of 83 sites name a sitemap. A robots.txt with no sitemap line throws away a free crawl hint. Check that yours lists every sitemap you publish.
AI39 of 83 sites name an AI crawler. Naming a crawler is not the same as blocking it. Some of those groups carry an allow rule. Decide what you want for each crawler and say it with the exact user agent.
UNKNOWN20 sites ship lines Google ignores. Unknown directives turn up across the census, from crawl-delay to stray HTML. The parser counts them and moves on. You should not assume they do anything.

The census is a fixed list of large, well resourced sites. It says nothing about the average site. Treat it as a description of how the biggest hosts behave, not as a benchmark for yours.

Check your own file

Run your robots.txt through the same parser

The checker on this site applies the same RFC 9309 semantics. Paste your file, pick a crawler, test any path, and it names the rule that decided. Use it to confirm what you actually block before you trust a big or messy file.

Then run your sitemap through the sitemap validator to confirm every URL you list is reachable.

Open the robots txt checker

05Questions about the robots.txt census

ANSWERWhat share of websites ship a valid robots.txt? Every one of the 83 hosts that served a body shipped a file that parsed cleanly under RFC 9309. None exceeded Google's 500 KiB limit. The only failures were the 10 of 93 hosts that returned no body, and they are excluded from every figure.
ANSWERHow was the census done? A fixed list of 93 hosts was fetched over HTTPS at /robots.txt with a 15 second timeout. The 83 with a body were parsed with a matcher implementing RFC 9309 and Google's open source parser. Non-standard lines were counted as unknown directives.
ANSWERWhy does the biggest file matter? Figma's file is 380,638 bytes with 8,823 rules. Google stops reading past 500 KiB. None of the 83 hosts hit that limit, but big files cost crawl budget and are harder to maintain.
ANSWERIs this a sample of the whole web? No. It is a fixed list of 93 well known hosts, not a random sample. It describes large, well resourced sites and says nothing about the average site. It is a single snapshot from 2026-08-24.