Everything You Want to Know about CCBot
CCBot is the crawler operated by the Common Crawl Foundation, building the free, open corpus of web crawl data used by researchers and to train many AI models.
Last verified August 15, 2026 against Common Crawl Foundation's published crawler docs
Why it's crawling your site
CCBot crawls your site to include it in Common Crawl's open dataset, which many organizations, including AI labs, use as training and research data.
How CCBot views your website
CCBot fetches raw page HTML (full page html, no js rendering; http/1.1 and http/2) — it does not execute JavaScript. Content that only appears after client-side rendering is invisible to it.
Paste a URL for the rendered-vs-crawled diff, blocked resources, and the sections that never reach the bot.
How to block CCBot
Add the rule below to your site's robots.txt. Most crawlers, including CCBot, check for changes within 24 hours.
User-agent: CCBot
Disallow: /Need the full walkthrough?
Partial blocks, per-directory rules, Cloudflare and server-level options, verification.