Crawler reference / robots.txt
OAI-SearchBot, GPTBot and ChatGPT-User: which one do you allow?
Each user agent below has a separate job. The table uses the vendor's own wording. The robots.txt block follows the table, and every claim carries the vendor page it came from.
| User agent | Job | What the vendor says it does | Effect of blocking |
|---|---|---|---|
OAI-SearchBot | Search | Surfaces websites in ChatGPT search results | Opted-out sites are not shown in search answers, but can still appear as navigational links |
GPTBot | Training | Collects web content that may train OpenAI foundation models | Signals that future material should be excluded from training data |
ChatGPT-User | User action | Visits a page because a ChatGPT user asked about it | The fetch is user-initiated, so robots.txt rules may not apply |
Why the three settings are independent
OpenAI documents the three tags as separate switches. Its crawler overview says each setting is independent of the others, and that a site owner can allow OAI-SearchBot to appear in search results while disallowing GPTBot to keep content out of training. When both are allowed, OpenAI says it may reuse the results of one crawl for both purposes to avoid duplicate crawling.
The practical consequence is that blocking GPTBot does not remove a page from ChatGPT search, and allowing OAI-SearchBot does not make a page training data.
What the vendor says about ChatGPT-User
OpenAI states that ChatGPT-User is not used to crawl the web automatically, and that because the action is started by a user, robots.txt rules may not apply. It also states that ChatGPT-User is not used to decide whether content appears in Search, and points site owners to OAI-SearchBot for search opt-outs.
Do not treat a ChatGPT-User allow rule as a search visibility switch. It is not one.
The robots.txt block
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Allow: /This block allows search, blocks training, and leaves user-initiated fetches alone. Keep the first group if you want ChatGPT search to be able to cite the page. Change only the GPTBot group if your position on training data changes.
How to check the result
Fetch the URL with each user agent and read the status. Use the robots.txt checker to test one URL against the rules, and the user agent parser to see what a request string claims to be.
Sources
Checked 2026-10-28 against the vendor page. Vendor wording can change; re-read the source before you copy a rule.
