robots.txt for Agents
How robots.txt controls AI agents
robots.txt (RFC 9309) is a voluntary-compliance protocol: you publish rules, and well-run crawlers choose to follow them. Nothing in the protocol enforces anything — which is why knowing which AI bots honor it matters as much as writing the rules.
Every major AI vendor now operates several bots with different jobs, and robots.txt treats each User-agent token as a separate addressee: a crawler obeys the most specific group matching its own token and ignores groups aimed at other tokens. Blocking one bot therefore does nothing to its siblings. The tokens sort into three tiers:
- Training crawlers (GPTBot, ClaudeBot, meta-externalagent) collect content for model training. Blocking one opts your content out of future models — it does not affect live answers or search citations.
- Search-index crawlers (OAI-SearchBot, Claude-SearchBot, PerplexityBot) build the indexes AI assistants cite. Allowing these is what makes your pages retrievable and citable in ChatGPT, Claude, and Perplexity answers.
- User-initiated fetchers (ChatGPT-User, Claude-User, Perplexity-User, Google-Agent) fetch a page live because a human just asked the assistant about it. Most vendors state this tier may ignore robots.txt — their reasoning: a person, not a crawler, requested the specific page. (Documented exceptions exist — Anthropic's Claude-User and Mistral's MistralAI-User both state they respect robots.txt.)
The practical consequence: User-agent: GPTBot + Disallow: / opts you out of OpenAI training but does not remove you from ChatGPT search (that's OAI-SearchBot) and does not stop live user fetches (that's ChatGPT-User). Each behavior needs its own group.
Current AI crawler tokens
| Token | Operator | Tier | Honors robots.txt? |
|---|---|---|---|
| GPTBot | OpenAI | Training | Yes |
| OAI-SearchBot | OpenAI | Search index | Yes — OpenAI recommends allowing it |
| ChatGPT-User | OpenAI | User fetch | "Rules may not apply" (OpenAI's wording) |
| ClaudeBot | Anthropic | Training | Yes |
| Claude-SearchBot | Anthropic | Search index | Yes |
| Claude-User | Anthropic | User fetch | Yes — a documented exception |
| PerplexityBot | Perplexity | Search index | Yes |
| Perplexity-User | Perplexity | User fetch | "Generally ignores" (Perplexity's wording) |
| Google-Extended | Training policy token | Token only — read by Googlebot, no crawler of its own | |
| Google-Agent | User fetch | Generally ignores | |
| meta-externalagent | Meta | Training | Yes |
Older lists still circulate anthropic-ai and claude-web — Anthropic no longer crawls under those tokens; its current crawlers are the three above. Each token in the table has a full page — exact UA string, robots.txt snippets, IP-range verification — in the AI agent user-agent reference.
Welcome agents explicitly
An explicit Allow group is a signal, not a no-op: the default is already "allowed", but naming an agent token tells crawl operators (and scanners like AgentGrade) that access is a decision, not an oversight.
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
Sitemap: https://example.com/sitemap.xml
The Sitemap: directive matters for agents too — it is how a crawler finds your canonical page list without walking every link.
Opt out of training, stay citable
The most common middle position: keep your pages citable in AI search while opting out of model training.
# Block training crawlers
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Google-Extended
Disallow: /
# Everything else (search indexers, user fetchers) stays allowed by default
Blocking everything — and what it costs
Disallowing every AI token makes your site invisible to AI search indexes, so assistants cannot cite you. It still does not firmly stop user-initiated fetchers — most of that tier ignores robots.txt by design. A firm block requires server-side controls: filtering by the vendors' published IP ranges or a WAF rule. Verification endpoints for each bot are listed on the /agents pages.
FAQ
Does blocking GPTBot remove my site from ChatGPT?
No. GPTBot only collects training data. ChatGPT search citations come from OAI-SearchBot's index, and live page visits come from ChatGPT-User — both keep working unless you block their tokens too (and ChatGPT-User may ignore the block).
Do AI agents actually respect robots.txt?
By tier: training and search crawlers from OpenAI, Anthropic, Perplexity, Google, and Meta document that they do. User-initiated fetchers mostly document that they do not (Claude-User and MistralAI-User are the stated exceptions). robots.txt is a published preference, not an access control.
Is Google-Extended a crawler I will see in my logs?
No. Google-Extended is a robots.txt policy token read by regular Googlebot. Disallowing it opts your content out of Gemini training and grounding without affecting Google Search. No request ever carries a Google-Extended user-agent.
Should my site allow or block AI crawlers?
It is a business decision about two separate things: training (your content improving future models) and visibility (assistants citing and recommending you). Many sites now block training tokens and allow search tokens — the "opt out of training, stay citable" pattern above.
Spec maturity
Established standard. robots.txt is specified in RFC 9309. The AI-specific tokens are defined and documented by each vendor, and change often enough that the user-agent reference is kept current per bot.
Learn more
- RFC 9309 — the robots.txt specification
- AI agent user-agents: the reference list — UA strings, robots.txt behavior, and IP verification per bot
Related
- llms.txt — the additive counterpart: robots.txt says what agents may fetch; llms.txt tells them where to start
- How AI agents actually browse the web