Everything You Want to Know about Claude Code
Claude Code is Anthropic's agentic coding tool; during a developer's session it can fetch web pages on demand (e.g. to read documentation) as part of completing a coding task.
Last verified August 15, 2026 against Anthropic's published crawler docs
Why it's crawling your site
A developer's Claude Code session requested a page directly; this is on-demand, user-directed access rather than automatic crawling.
How Claude Code views your website
Claude Code fetches raw page HTML (on-demand single fetch, agent-directed) — it does not execute JavaScript. Content that only appears after client-side rendering is invisible to it.
Paste a URL for the rendered-vs-crawled diff, blocked resources, and the sections that never reach the bot.
How to block Claude Code
Add the rule below to your site's robots.txt. Most crawlers, including Claude Code, check for changes within 24 hours.
User-agent: Claude-Code
Disallow: /Need the full walkthrough?
Partial blocks, per-directory rules, Cloudflare and server-level options, verification.