Everything You Want to Know about GPTBot
GPTBot is OpenAI's web crawler used to collect publicly available content that may be used to train OpenAI's generative AI foundation models. Disallowing GPTBot signals that a site's content should be excluded from training.
Last verified August 15, 2026 against OpenAI's published crawler docs
Why it's crawling your site
GPTBot crawls your site to gather content OpenAI may use to train future generations of its generative AI models. It does not affect ChatGPT's search results or user-triggered fetches, which use OAI-SearchBot and ChatGPT-User respectively.
How GPTBot views your website
GPTBot fetches raw page HTML (full page html, no js rendering documented) — it does not execute JavaScript. Content that only appears after client-side rendering is invisible to it.
Paste a URL for the rendered-vs-crawled diff, blocked resources, and the sections that never reach the bot.
How to block GPTBot
Add the rule below to your site's robots.txt. Most crawlers, including GPTBot, check for changes within 24 hours.
User-agent: GPTBot
Disallow: /Need the full walkthrough?
Partial blocks, per-directory rules, Cloudflare and server-level options, verification.