Reference
robots.txt matches on the user-agent token, not on the company name — so
GPTBot and OAI-SearchBot are two different switches for the same vendor,
with two very different consequences. This is the list we check against, grouped by what
happens when you block one.
These decide whether your pages can appear in AI answers. Blocking one removes you from its answers — no error, no warning, you simply stop being cited.
| Crawler | User-agent token | What it does | Source |
|---|---|---|---|
| OAI-SearchBot | oai-searchbot | ChatGPT's search index. Sites blocked here are not shown in ChatGPT search answers. | OpenAI docs |
| Claude-SearchBot | claude-searchbot | Claude's search index. Disabling it may reduce your visibility in search responses. | Anthropic docs |
| PerplexityBot | perplexitybot | Surfaces and links sites in Perplexity results. Perplexity states it is not used to train foundation models. | Perplexity docs |
| Bingbot | bingbot | Bing's index, which also feeds Copilot. | Bing docs |
These feed model training. Restricting them lowers your training exposure and does not change whether you get cited — which is why mixing them up with the group above is costly.
| Crawler | User-agent token | What it does | Source |
|---|---|---|---|
| GPTBot | gptbot | OpenAI model training. Blocking it signals your content should not be used for training. | OpenAI docs |
| ClaudeBot | claudebot | Anthropic model training. | Anthropic docs |
| Google-Extended | google-extended | Gemini model training. Google states it has no effect on Search ranking. | Google docs |
These fetch a page because a person asked a question right then, not on a schedule. OpenAI states robots.txt may not apply to them; we list them so they don't get mistaken for the crawlers that run your citation fate.
| Crawler | User-agent token | What it does | Source |
|---|---|---|---|
| ChatGPT-User | chatgpt-user | Triggered when someone asks ChatGPT. OpenAI notes robots.txt may not apply to it. | OpenAI docs |
| Claude-User | claude-user | Triggered when someone asks Claude. | Anthropic docs |
| Perplexity-User | perplexity-user | Triggered when someone asks Perplexity. | Perplexity docs |
Blocking a retrieval crawler quietly removes you from AI answers. There is no error, no notification, nothing in your logs except a request you never got — you find out when somebody notices you never come up.
Blocking a training crawler does none of that. It lowers your training exposure
and leaves your citations untouched. Plenty of site owners block GPTBot believing
it buys them something in search visibility; by OpenAI's own documentation it does not — that
switch is OAI-SearchBot. The reverse mistake is just as easy, and worse: allowing
everything because you assumed "AI crawler" meant "training crawler".
Because it would go stale and you would not find out. Vendors publish their current ranges themselves, and they change them — the fact that OpenAI and Perplexity publish these as machine-readable files (linked from each vendor's docs above) is a hint about how often they move. If you are verifying traffic in your logs, check the vendor's published range rather than a copy of it on someone else's page. Ours included.
One more caveat worth stating plainly: robots.txt is a request, not a lock. The
major vendors honour it — Anthropic's docs say so explicitly, including support for
Crawl-delay — but a crawler that ignores it is a matter between you and the vendor,
and a real block belongs at your edge, not in a text file.
Knowing the tokens is half of it; the other half is what your own robots.txt
actually says about them. Run the checker and it will report
which of these are blocked, with the evidence. If you want to change the policy,
generate a file that separates the two groups for you.