Everything You Want to Know about Claude-User
Claude-User supports Claude AI users: when someone asks Claude a question, it may access a website using the Claude-User agent to help answer it.
Last verified August 15, 2026 against Anthropic's published crawler docs
Why it's crawling your site
A user asked Claude a question that required fetching your page; disabling Claude-User prevents retrieval in response to user queries, which may reduce visibility for user-directed web search.
How Claude-User views your website
Claude-User fetches raw page HTML (single-page fetch, user-triggered) — it does not execute JavaScript. Content that only appears after client-side rendering is invisible to it.
Paste a URL for the rendered-vs-crawled diff, blocked resources, and the sections that never reach the bot.
How to block Claude-User
Add the rule below to your site's robots.txt. Most crawlers, including Claude-User, check for changes within 24 hours.
User-agent: Claude-User
Disallow: /Need the full walkthrough?
Partial blocks, per-directory rules, Cloudflare and server-level options, verification.