AI agent user-agents: the reference list
Updated 2026-07-19
The major AI crawlers and fetchers, one page each: the exact user-agent string, whether it respects robots.txt, how to allow or block it, and how to verify the traffic is genuine. The key distinction is the three-tier system every vendor runs — training crawlers (collect content for model training, respect robots.txt), search-index crawlers (build the index behind cited answers, respect robots.txt), and user-initiated fetchers (fetch live because a human asked; most vendors state these ignore robots.txt). Blocking one tier's token does nothing to the others — the robots.txt guide explains why. Strictly speaking, most of these are crawlers and fetchers that serve agent systems rather than agents themselves — the User-Agent string identifies the fetching component, never the whole agent. What is an AI agent? covers the distinction.
| Bot | Operator | Tier | Honors robots.txt? |
|---|---|---|---|
| ClaudeBot | Anthropic | Training crawler | Yes |
| Claude-User | Anthropic | User-initiated fetcher | Yes (documented exception) |
| Claude-SearchBot | Anthropic | Search-index crawler | Yes |
| GPTBot | OpenAI | Training crawler | Yes |
| OAI-SearchBot | OpenAI | Search-index crawler | Yes |
| ChatGPT-User | OpenAI | User-initiated fetcher | May ignore (vendor-documented) |
| PerplexityBot | Perplexity | Search-index crawler | Yes |
| Perplexity-User | Perplexity | User-initiated fetcher | May ignore (vendor-documented) |
| Google-Extended | Training crawler | Token only — no crawler of its own | |
| Google-Agent | User-initiated fetcher | May ignore (vendor-documented) | |
| Google-CloudVertexBot | Owner-commissioned crawler | Yes | |
| Meta-ExternalAgent | Meta | Training crawler | Yes |
| Meta-ExternalFetcher | Meta | User-initiated fetcher | May ignore (vendor-documented) |
| MistralAI-User | Mistral AI | User-initiated fetcher | Yes (documented exception) |
| Applebot | Apple | Search-index crawler | Yes |
| Applebot-Extended | Apple | Training crawler | Token only — no crawler of its own |
| Amazonbot | Amazon | Training crawler | Yes |
| bingbot | Microsoft | Search-index crawler | Yes |
| DuckAssistBot | DuckDuckGo | Search-index crawler | Yes |
| CCBot | Common Crawl | Training crawler | Yes |
| Bytespider | ByteDance | Training crawler | Unverified — compliance contested |
Allow the well-behaved tiers (copy-paste)
Explicitly welcomes every documented training and search crawler; user-initiated fetchers need no rule (they fetch on a human's request).
User-agent: ClaudeBot
Allow: /
User-agent: Claude-SearchBot
Allow: /
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: meta-externalagent
Allow: /
User-agent: Applebot
Allow: /
User-agent: Amazonbot
Allow: /
User-agent: bingbot
Allow: /
User-agent: DuckAssistBot
Allow: /
User-agent: CCBot
Allow: /
Sitemap: https://example.com/sitemap.xmlBlock training, stay citable (copy-paste)
Opts out of model training while leaving AI search indexing — the tier that makes you citable in answers — untouched.
User-agent: ClaudeBot
Disallow: /
User-agent: GPTBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: meta-externalagent
Disallow: /
User-agent: Applebot-Extended
Disallow: /
User-agent: Amazonbot
Disallow: /
User-agent: CCBot
Disallow: /Scan your site to see how it treats these agents, build your policy with the robots.txt generator, or read the robots.txt guide and how AI agents browse.