Crawler reference / robots.txt
ClaudeBot, Claude-SearchBot and Claude-User: which one does what?
Each user agent below has a separate job. The table uses the vendor's own wording. The robots.txt block follows the table, and every claim carries the vendor page it came from.
| User agent | Job | What the vendor says it does | Effect of blocking |
|---|---|---|---|
ClaudeBot | Training | Collects web content that could contribute to model training | Future material should be excluded from training datasets |
Claude-SearchBot | Search index | Analyses content to improve the relevance of search responses | The content is not indexed for search optimisation |
Claude-User | User action | Accesses a page for a user-initiated Claude request | The system cannot retrieve the content for a user query |
Three tags, three jobs
Anthropic's support article lists three robots and the effect of disabling each one. ClaudeBot is the training crawler. Claude-SearchBot navigates the web to improve search result quality. Claude-User supports user requests and controls which sites those requests can reach.
Because the switches are separate, a site can allow search and refuse training. That is the same split OpenAI documents for OAI-SearchBot and GPTBot.
What Anthropic says about robots.txt
The same article states that Anthropic bots respect industry-standard directives in robots.txt, and that they do not try to bypass CAPTCHAs. It also states that Anthropic supports the non-standard Crawl-delay extension, with a one-second example for ClaudeBot.
Crawl-delay is not part of the robots.txt standard and other crawlers may ignore it. Treat it as a request to Anthropic, not a general rate limit.
The robots.txt block
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: Claude-User
Allow: /
User-agent: ClaudeBot
Crawl-delay: 1Do not write two ClaudeBot groups in one file. Merge the Crawl-delay line into the existing ClaudeBot group instead, because a second group for the same agent is easy to misread and easy to break in review.
How to check the result
Fetch the page with each of the three user agent strings and record the status. The robots.txt checker tests a URL against the rules, and the user agent parser shows what a request claims to be.
Sources
Checked 2026-10-28 against the vendor page. Vendor wording can change; re-read the source before you copy a rule.
