Free tool
Enter a URL. We fetch it the way an AI crawler does — no JavaScript, no cookies — and report what is actually there: how much readable text, whether the crawlers that feed AI answers are blocked, and what the page tells machines about itself. Every finding carries its evidence.
Anonymous checks are limited per day. We respect robots.txt and identify
ourselves in the User-Agent.
They are not interchangeable: blocking a retrieval crawler removes you from citations, while restricting a training crawler costs you nothing in visibility. Getting this backwards is the most common mistake we see — and it usually goes in the direction of blocking the wrong one.
These decide whether your pages can appear in AI answers. Blocking one removes you from its answers — no error, no warning, you simply stop being cited.
| Crawler | User-agent token | What it does | Source |
|---|---|---|---|
| OAI-SearchBot | oai-searchbot | ChatGPT's search index. Sites blocked here are not shown in ChatGPT search answers. | OpenAI docs |
| Claude-SearchBot | claude-searchbot | Claude's search index. Disabling it may reduce your visibility in search responses. | Anthropic docs |
| PerplexityBot | perplexitybot | Surfaces and links sites in Perplexity results. Perplexity states it is not used to train foundation models. | Perplexity docs |
| Bingbot | bingbot | Bing's index, which also feeds Copilot. | Bing docs |
These feed model training. Restricting them lowers your training exposure and does not change whether you get cited — which is why mixing them up with the group above is costly.
| Crawler | User-agent token | What it does | Source |
|---|---|---|---|
| GPTBot | gptbot | OpenAI model training. Blocking it signals your content should not be used for training. | OpenAI docs |
| ClaudeBot | claudebot | Anthropic model training. | Anthropic docs |
| Google-Extended | google-extended | Gemini model training. Google states it has no effect on Search ranking. | Google docs |
These fetch a page because a person asked a question right then, not on a schedule. OpenAI states robots.txt may not apply to them; we list them so they don't get mistaken for the crawlers that run your citation fate.
| Crawler | User-agent token | What it does | Source |
|---|---|---|---|
| ChatGPT-User | chatgpt-user | Triggered when someone asks ChatGPT. OpenAI notes robots.txt may not apply to it. | OpenAI docs |
| Claude-User | claude-user | Triggered when someone asks Claude. | Anthropic docs |
| Perplexity-User | perplexity-user | Triggered when someone asks Perplexity. | Perplexity docs |
Googlebot is deliberately absent. It indexes you for search the way it always has; what decides whether an AI system can use your page is whether the page is readable without JavaScript, and whether the crawlers above are allowed in. Every row links to the vendor's own documentation — the full user-agent reference is here.
The major ones do, with a caveat that matters: respect is voluntary, and it varies. What makes
it worth getting right is that robots.txt is the one place where a single line
silently decides whether you can be cited at all — and unlike a rendering problem, a blocked
crawler produces no error anywhere. You only find out when someone notices you never come up.
Because you are looking at it with your browser's JavaScript running. Most AI crawlers don't run JavaScript. On a client-rendered page, the two of you are looking at genuinely different documents — yours has the content, theirs is empty. That gap is what this checker measures, and it is the failure people are most surprised by.
The report ranks findings by severity and attaches the raw evidence — status codes, byte counts,
the exact snippet — so you can reproduce every conclusion with curl. Where a file
fixes it, you get the file: a corrected robots.txt, an llms.txt, JSON-LD for your pages. The
full list of checks is public, including what each one looks for.
If you are auditing a site for someone else rather than for yourself, the same check is written up in those terms on the AI SEO audit page — including what the audit deliberately leaves out.