Everything You Want to Know about Bytespider
Bytespider is ByteDance's general-purpose crawler, used to gather web content for training ByteDance's AI models, including the Doubao family.
Last verified August 15, 2026 against ByteDance's published crawler docs
Why it's crawling your site
Bytespider crawls your site to collect training data for ByteDance's AI models; ByteDance states the bot honors robots.txt, though independent log analyses have reported inconsistent compliance historically.
How Bytespider views your website
Bytespider fetches raw page HTML (full page html) — it does not execute JavaScript. Content that only appears after client-side rendering is invisible to it.
Paste a URL for the rendered-vs-crawled diff, blocked resources, and the sections that never reach the bot.
How to block Bytespider
Add the rule below to your site's robots.txt. Most crawlers, including Bytespider, check for changes within 24 hours.
User-agent: Bytespider
Disallow: /Need the full walkthrough?
Partial blocks, per-directory rules, Cloudflare and server-level options, verification.