Everything You Want to Know about Claude-SearchBot
Claude-SearchBot navigates the web to improve the relevance and accuracy of Claude's search-style results.
Last verified August 15, 2026 against Anthropic's published crawler docs
Why it's crawling your site
Anthropic crawls your site with Claude-SearchBot to index content for Claude's search features; disabling it may reduce your site's visibility and accuracy in user search results within Claude.
How Claude-SearchBot views your website
Claude-SearchBot fetches raw page HTML (full page html) — it does not execute JavaScript. Content that only appears after client-side rendering is invisible to it.
Paste a URL for the rendered-vs-crawled diff, blocked resources, and the sections that never reach the bot.
How to block Claude-SearchBot
Add the rule below to your site's robots.txt. Most crawlers, including Claude-SearchBot, check for changes within 24 hours.
User-agent: Claude-SearchBot
Disallow: /Need the full walkthrough?
Partial blocks, per-directory rules, Cloudflare and server-level options, verification.