Everything You Want to Know about ClaudeBot
ClaudeBot is Anthropic's web crawler that collects publicly available content that could contribute to training Anthropic's generative AI models.
Last verified August 15, 2026 against Anthropic's published crawler docs
Why it's crawling your site
ClaudeBot crawls your site to gather content for Anthropic's AI model training datasets; disallowing it excludes future content from that training.
How ClaudeBot views your website
ClaudeBot fetches raw page HTML (full page html) — it does not execute JavaScript. Content that only appears after client-side rendering is invisible to it.
Paste a URL for the rendered-vs-crawled diff, blocked resources, and the sections that never reach the bot.
How to block ClaudeBot
Add the rule below to your site's robots.txt. Most crawlers, including ClaudeBot, check for changes within 24 hours.
User-agent: ClaudeBot
Disallow: /Need the full walkthrough?
Partial blocks, per-directory rules, Cloudflare and server-level options, verification.