Comparison
They are not two ways of doing the same thing. robots.txt gives permission —
it decides whether a crawler may read you. llms.txt gives a map — it tells
whoever reads it what matters. A map cannot grant permission, and a permission file
says nothing about what your site is about.
robots.txt | llms.txt | sitemap.xml | JSON-LD | |
|---|---|---|---|---|
| What it is | Permission rules, per crawler | A short map of what matters | A list of your URLs | Facts declared inside a page |
| Who reads it | Crawlers, before they fetch anything | Anything that chooses to | Crawlers, to find pages they missed | Anything that parses the page |
| If it's ignored | Your block isn't honoured — a real block belongs at your edge | Nothing breaks; you just don't get the hint | Deep pages get found more slowly | Machines infer instead of knowing |
| Status | Long-standing standard | Emerging convention, no obligation | Long-standing standard | Long-standing standard |
| Where | /robots.txt | /llms.txt | /sitemap.xml | In the <head> |
Someone blocks an AI crawler in robots.txt, then adds an llms.txt
expecting it to bring the crawler back. It cannot. Nothing in llms.txt overrides a
Disallow — the crawler never gets as far as reading your map, because it was told
not to come in. If you want to be cited, the change has to happen in
robots.txt.
The trap is that it fails silently and in the flattering direction: the site owner sees a neat
llms.txt, assumes the AI part is handled, and never learns that the crawler that
decides citations is blocked. You can see the whole split on
the user-agent list — which crawlers bring citations and
which only feed training.
Yes, with honest expectations: it is cheap, it is a hint, and no model is obligated to read it. What it can do is remove guesswork about which of your thirty pages actually matter. What it cannot do is substitute for the two things that decide whether you are readable at all — being allowed in, and having your content present in the static HTML.
It solves a different problem again: discovery. robots.txt says who may come in,
sitemap.xml says where the pages are, llms.txt says which ones matter,
JSON-LD says what they mean. A site can be fully crawlable and still be a mystery — that's the
gap the last two close.
robots.txt generator — keeps retrieval crawlers and
training crawlers in separate groups, so blocking one doesn't cost you the other.
llms.txt generator — a map with your name, one line about the
site, and the pages that matter.
JSON-LD generator — the Organization and WebSite declarations that
tell machines who you are.
Then check it actually works: the AI crawler checker fetches your page the way a crawler does and reports what it found, with the evidence behind every finding. A generated file is a hypothesis until you test it.