ClaudeBot

Anthropic · Training crawler · Updated 2026-07-19

ClaudeBot is Anthropic’s training crawler — it crawls the web to collect content for model training. Blocking it opts your content out of future models but does not affect live answers.

User-agent string

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; ClaudeBot/1.0; +claudebot@anthropic.com)

Source: community-documented (Anthropic describes the bot but does not publish the full string).

Does ClaudeBot respect robots.txt?

Respects robots.txt, per Anthropic’s crawler documentation.

Allow or block ClaudeBot

# Allow (recommended if you want ClaudeBot traffic)
User-agent: ClaudeBot
Allow: /

# Block
User-agent: ClaudeBot
Disallow: /

Verifying it’s really ClaudeBot

Anthropic publishes crawler IP ranges at claude.com/crawling/bots.json (216.73.216.0/22 plus individual addresses). A ClaudeBot user-agent from an IP outside those ranges is an impersonator. User-agent strings can be spoofed by anyone — identity claims are only trustworthy when the source IP matches the operator’s published ranges.

Fetch the published ranges and check a suspect IP against them:

curl -s https://claude.com/crawling/bots.json

What site owners should know

ClaudeBot collects web content for training Anthropic’s models. It is distinct from Claude-User (live fetches for users) and Claude-SearchBot (search indexing) — blocking ClaudeBot does not block the other two.

Scan your site to see how your robots.txt, content negotiation, and discovery files treat AI agents — including whether you’re blocking agents you meant to allow. Background: the robots.txt guide, how AI agents browse, and agent readiness.

All AI agent user-agents

ClaudeBot · Claude-User · Claude-SearchBot · GPTBot · OAI-SearchBot · ChatGPT-User · PerplexityBot · Perplexity-User · Google-Extended · Google-Agent · Google-CloudVertexBot · Meta-ExternalAgent · Meta-ExternalFetcher · MistralAI-User · Applebot · Applebot-Extended · Amazonbot · bingbot · DuckAssistBot · CCBot · Bytespider