Everything You Want to Know about AI2Bot
AI2Bot is the Allen Institute for AI's crawler, used to collect web content for building open-source AI research datasets and training open language models.
Last verified August 15, 2026 against Allen Institute for AI (AI2)'s published crawler docs
Why it's crawling your site
AI2Bot crawls your site to gather content for AI2's open training datasets; it honors robots.txt.
How AI2Bot views your website
AI2Bot fetches raw page HTML (full page html) — it does not execute JavaScript. Content that only appears after client-side rendering is invisible to it.
Paste a URL for the rendered-vs-crawled diff, blocked resources, and the sections that never reach the bot.
How to block AI2Bot
Add the rule below to your site's robots.txt. Most crawlers, including AI2Bot, check for changes within 24 hours.
User-agent: AI2Bot
Disallow: /Need the full walkthrough?
Partial blocks, per-directory rules, Cloudflare and server-level options, verification.