Everything You Want to Know about Amazon Q Business
The Amazon Q Business Web Crawler indexes website content that customers configure Amazon Q Business to crawl, powering generative-AI answers grounded in that content.
Last verified August 15, 2026 against Amazon's published crawler docs
Why it's crawling your site
A company using Amazon Q Business configured your site as a data source; the crawler fetches and indexes your pages to ground the assistant's answers.
How Amazon Q Business views your website
Amazon Q Business fetches raw page HTML (full page html) — it does not execute JavaScript. Content that only appears after client-side rendering is invisible to it.
Paste a URL for the rendered-vs-crawled diff, blocked resources, and the sections that never reach the bot.
How to block Amazon Q Business
Add the rule below to your site's robots.txt. Most crawlers, including Amazon Q Business, check for changes within 24 hours.
User-agent: amazon-QBusiness
Disallow: /Need the full walkthrough?
Partial blocks, per-directory rules, Cloudflare and server-level options, verification.