Everything You Want to Know about AhrefsBot
AhrefsBot is Ahrefs' primary web crawler, continuously crawling the web to build and update its backlink database powering Ahrefs' SEO tools.
Last verified August 15, 2026 against Ahrefs's published crawler docs
Why it's crawling your site
AhrefsBot crawls your site (and pages linking to it) to keep Ahrefs' backlink index current; it strictly respects robots.txt, including crawl-delay.
How AhrefsBot views your website
AhrefsBot fetches raw page HTML (full page html) — it does not execute JavaScript. Content that only appears after client-side rendering is invisible to it.
Paste a URL for the rendered-vs-crawled diff, blocked resources, and the sections that never reach the bot.
How to block AhrefsBot
Add the rule below to your site's robots.txt. Most crawlers, including AhrefsBot, check for changes within 24 hours.
User-agent: AhrefsBot
Disallow: /Need the full walkthrough?
Partial blocks, per-directory rules, Cloudflare and server-level options, verification.