Free tool
One file decides whether AI systems can reach your pages at all. The rule that matters most is not which company you block — it is whether you are blocking a crawler that brings you citations or one that only feeds model training. This generator keeps those two groups apart.
We leave this empty by default. Deciding which paths of a site you don't know should be hidden is not ours to make — and a rule for a path that doesn't exist makes people think something is protected when it isn't.
Enter your site address above — the file appears here as you type.
A single User-agent: * would be shorter. But a crawler with its own group
does not inherit the wildcard rules — which cuts both ways. If you list a
retrieval crawler to make sure it's allowed, you have to repeat your private-path rules inside
that group, or you have accidentally opened those paths to it. This file writes every group out
in full for that reason, and you can see the whole policy at a glance instead of inferring it.
The major vendors do, and their documentation says so — Anthropic even supports the non-standard
Crawl-delay. But treat it as a request rather than an access control: it is
voluntary, it is per-crawler, and it says nothing about what has already been collected. If you
need a guarantee, it belongs at your edge or origin, not in a text file.
A generated file is a hypothesis until you test it. Run the AI crawler checker on the same address and it will report which crawlers are blocked, how much text is readable without JavaScript, and the raw evidence behind each finding. The full user-agent list explains what each token does, with links to the vendor docs.