AI crawlers and robots.txt: what each user-agent actually controls

Most sites inherited their robots.txt from a template and have never read it. Since 2023 that file also decides which AI systems may read the site, so it is worth ten minutes of attention.

First: robots.txt is a request, not a lock

The Robots Exclusion Protocol is voluntary. It works because the large operators choose to honour it and publish their user-agents and IP ranges so you can verify who is who. A crawler that ignores it is not breaking a technical control — there is no technical control. If you need enforcement, that lives at the server or CDN layer: rate limits, firewall rules, authentication. Those actually refuse the request.

The main AI user-agents

Two of those entries are not crawlers at all. Google-Extended and Applebot-Extended are control tokens: you address them in robots.txt, but no bot fetches your site under that name. They modify what the operator is permitted to do with pages fetched by its ordinary crawler.

The Google-Extended misunderstanding

This one is worth stating plainly, because it is the most common mistake in the whole topic. Google-Extended governs whether your content may be used for Gemini and for Vertex AI grounding. It does not affect whether Googlebot indexes your site, and it does not remove you from AI features that are built on the Google Search index.

So a site owner who wants out of "Google's AI" and adds a Google-Extended disallow has not achieved that, and a site owner who fears that adding it will hurt their Search rankings is worrying about the wrong thing. The only way to leave AI features grounded in Search is to leave Search — which is a much larger decision.

Three things robots.txt cannot do

1. It cannot remove a page from search results

Disallow stops a compliant crawler from fetching a URL. It does not stop that URL from being listed if other sites link to it — the engine simply lists it without a description, having never read it. To keep a page out of results you need a noindex directive, and the crawler must be allowed to fetch the page in order to see the tag. Blocking and noindexing the same URL is self-defeating.

2. It cannot reach into a model that is already trained

Blocking a training crawler affects future collection. Weights that already exist are not edited by a text file. Any expectation that adding a line today withdraws content from a shipped model is misplaced.

3. It cannot stop a crawler that does not care

Scrapers that ignore the protocol also tend to forge their user-agent string. Verification, where an operator supports it, is done by reverse DNS or by published IP ranges — not by trusting the header.

A worked example

The version below allows AI search and user fetches, declines training collection, and points to the sitemap. It is one defensible position among several — it is not a recommendation, because the trade-off is a business decision:

# Ordinary search engines
User-agent: *
Allow: /

# Training collection — declined
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

User-agent: Google-Extended
Disallow: /

# AI search and user-triggered fetches — allowed
User-agent: OAI-SearchBot
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: PerplexityBot
Allow: /

Sitemap: https://example.com/sitemap.xml

How to check what you actually have

• Open https://yourdomain.com/robots.txt in a browser. That is the file — there is no other copy.
• Use the robots.txt report in Google Search Console to see the version Google last fetched and any parse errors.
• Grep your server logs for the user-agents above to see which ones visit and what status codes they receive.

A file that has never been read is a decision that was never made. Reading it takes a minute, and the file is public either way.