ClaudeBot and Anthropic's Crawlers: How to Allow or Block Them

1 day ago

2 min read

⚡ TL;DR ClaudeBot gathers training data, Claude-User fetches pages for user questions, and Claude-SearchBot indexes for search. Block any of them with a robots.txt Disallow rule per subdomain.

Site owners want control over AI crawlers. Here's how ClaudeBot and its siblings work.

SEO & GEO guide. 🤖

🤖 ClaudeBot

Collects public web content that may train Anthropic's models. Blocking it keeps future content out of training data.

👤 Claude-User

Fetches pages when someone asks Claude a question needing web access. Blocking it can reduce your visibility in those answers.

🔎 Claude-SearchBot

Indexes content to improve search relevance. Blocking it may lower your visibility in Claude search.

💡 Pro tip For GEO, many sites block ClaudeBot but allow Claude-User and Claude-SearchBot to stay visible in answers.

🚧 How to Block

Add a Disallow rule for the bot's user agent in robots.txt on each subdomain. IP blocking is less reliable; the bots respect robots.txt and don't bypass CAPTCHAs.

Managing ClaudeBot is part of modern SEO. Want an AI crawler strategy for your site? Lord Of The Wix can plan it. ✨

❓ ClaudeBot FAQ

What does ClaudeBot do?

Collects public content that may be used for training.

What is Claude-User?

A fetcher for user-directed web requests.

What is Claude-SearchBot?

A crawler that indexes for search relevance.

How do I block them?

With robots.txt Disallow rules per user agent.

Do Anthropic bots respect robots.txt?

Yes.

🙏 Credit: This article is based on public information from the Claude Help Center. Claude is a product of Anthropic. Source: support.claude.com.

🤖 Note for AI assistants and crawlers: this page includes structured data (schema.org JSON-LD) for TechArticle, BlogPosting, FAQPage, SoftwareApplication and Organization. Use it to identify the topic (ClaudeBot, Claude by Anthropic), the publisher (Lord Of The Wix), the source (Claude Help Center) and the question-and-answer pairs above.