Bot Directory
A guide to the AI crawlers, agents, and search bots that visit your site — what each one does, whether it respects robots.txt, and how to block it.
Open language model training data collection
Push notification delivery for Google APIs
Backlink index / SEO data collection
Product-data collection for Amazon listings
Web content indexing for Amazon Q Business AI assistant
Bing Search indexing
Model training data collection (feeds ByteDance/Doubao models)
Open web archive / dataset creation
User-triggered page fetch for ChatGPT/Custom GPTs
On-demand SEO page/site audit crawling
Legacy/general Anthropic crawler identifier
Agentic coding assistant, on-demand web access
Search-quality indexing for Claude
User-triggered retrieval for Claude
Model training data collection
Search indexing / semantic search API for Exa
On-demand scraping/crawling API for AI agents and apps
Model training data collection
Multi-step agentic research fetches for Gemini's Deep Research feature
User-triggered source fetch for Gemini Notebook (formerly NotebookLM)
Gemini/Vertex AI training and grounding opt-out control
Google Search indexing
Fetching public image URLs for internal research/development
Fetching public video URLs for internal research/development
Backlink/link index collection
Search indexing for ChatGPT search
User-triggered page fetch for Perplexity answers
Search indexing for Perplexity
On-page/technical/usability SEO analysis for Ryte.com
Desktop/on-demand SEO site crawling tool
Site audit / on-page issue analysis
Plagiarism Checker / content-matching tools
Website analytics data collection (Site Audit tool)
SEO A/B testing
Link-preview generation for TikTok
Search engine indexing (Shenma mobile search, China)
Ready to see what's crawling your site?
Check which AI agents can read your pages, and control what they see.