User agent parser that can prove a crawler is real

01Read the claim, then check it

Paste the user agent string from your access log. If it claims to be a crawler that publishes its address ranges, add the IP from the same log line and the check runs against the published list. Everything happens in your browser and nothing is sent anywhere.

The address ranges are the files Google and Bing publish, stored beside this page with the timestamp each operator stamped on them. Nobody else publishes a machine readable range list, so for every other crawler this page reads the claim and then says plainly that it cannot verify it.

02A user agent string proves nothing on its own

The user agent is a header the client chooses. Setting it to Googlebot takes one line of code, and scrapers do it constantly, because a lot of sites wave through anything that says Googlebot. That is why blocking or allowing on the string alone is not a control, it is a suggestion.

Both operators that publish ranges say the same thing in their own documentation. Confirm the request by address, not by the header. Google gives two ways to do it and this page implements the one a browser can do honestly.

1. reverse DNS      PTR on the IP must end in googlebot.com or google.com
                    then a forward lookup on that name must return the same IP
                    a browser cannot do DNS, so this page cannot run it for you

2. published range  the IP must fall inside one of the prefixes in the JSON
                    files Google publishes. That is what the tool above runs.

The second check is the one that works from a static page, so it is the one implemented here. If you need the reverse DNS check as well, run host on the address and then run host on the name it returns, and confirm you get back the address you started with.

03What a match does and does not tell you

A match means the request came from an address the operator publishes as its own. That is a strong signal about origin and it is worth having. It is not a statement that the request is welcome, and there are three specific things it does not mean.

It does not mean the crawler will obey your robots.txt, though Google and Bing do. It does not mean the traffic is harmless, because a real crawler can still hammer a slow endpoint. And it does not stay true, because these lists change. Every file on this page carries the timestamp its operator stamped on it, shown next to the result, so a stale answer is visible rather than silent.

A non match is weaker evidence than a match. It can mean a spoof, and it can equally mean a crawler that simply does not publish ranges, which is most of them. The tool says which of those two it is rather than reporting every non match as a fake.

04The lists this page checks against

Four files, fetched on 2026-08-24 and republished here unmodified. The timestamps are the operators own, not ours.

Google splits its crawlers across three files on purpose. Googlebot is search crawling. Special crawlers cover things like AdsBot. User triggered fetchers cover requests a person set off, such as Search Console inspecting a URL, and that file is the largest of the three.

05Which crawlers real sites actually name

Identifying a crawler is more useful when you know whether anyone else treats it as worth naming. On 2026-08-24, 83 major sites returned a robots.txt, and these are the user agent tokens they name most often. A site is counted once per token however many times it appears.

The AI crawlers are the story in that table. 39 of 83 sites now name at least one, which was rare two years ago. Naming is not the same as blocking, since some of those groups carry an allow rule, and the count is a token match on the user agent line rather than a judgement about intent.

If you are writing rules for any of these, the robots txt checker on this site will tell you which rule actually wins for a given path, including the case where a group named for one crawler silently switches off the rules in the star group.

06The tests this parser passes

A tool that cannot show its own results is asking to be trusted rather than checked. All 59 of these run against the code on this page, and the verification cases run against the real published ranges rather than invented ones.

08Where the ranges come from

• DATAgooglebot.json, Google's own published Googlebot ranges.
• DATAspecial-crawlers.json, Google's ranges for AdsBot and similar.
• DATAuser-triggered-fetchers.json, Google's ranges for fetches a person set off.
• DATAbingbot.json, Microsoft's published Bingbot ranges.
• DATAThe reference file, all four lists plus the robots.txt token counts, exactly as this page reads them.