Everything You Want to Know about FirecrawlAgent
FirecrawlAgent is the user-agent Firecrawl uses when its scrape, crawl, or agent API fetches pages on behalf of customers building AI applications.
Last verified August 15, 2026 against Firecrawl's published crawler docs
Why it's crawling your site
A Firecrawl customer's application requested your page via the Firecrawl API (scrape, crawl, or agent job); Firecrawl respects robots.txt by default, though enterprise customers can request an override.
How FirecrawlAgent views your website
FirecrawlAgent renders JavaScript before reading your page, so client-side content, hydrated components, and JS-injected text are all visible to it, the same as a browser would see.
Paste a URL for the rendered-vs-crawled diff, blocked resources, and the sections that never reach the bot.
How to block FirecrawlAgent
Add the rule below to your site's robots.txt. Most crawlers, including FirecrawlAgent, check for changes within 24 hours.
User-agent: FirecrawlAgent
Disallow: /Need the full walkthrough?
Partial blocks, per-directory rules, Cloudflare and server-level options, verification.