AI crawler profile
What is Diffbot? How to allow or block Diffbot's crawler
Operator
Diffbot
Purpose
Training + search
robots.txt
Partially respects robots.txt
Official docs
Diffbot converts web pages into structured data (its Knowledge Graph) that customers — including AI companies — query and license. It's less a classic crawler than an extraction service: many fetches happen on demand when a customer requests a specific URL, which complicates the usual robots.txt picture.
Does Diffbot respect robots.txt?
Diffbot's bulk crawling honors robots.txt, but customer-triggered, on-demand extraction of specific URLs has been reported to proceed regardless — treat blocks as advisory.
Block Diffbot with robots.txt
User-agent: Diffbot Disallow: /
Explicitly allow Diffbot
User-agent: Diffbot Allow: /
Should you block Diffbot?
It's a genuine trade-off: Diffbotserves both search-style features (which send you visibility) and model training (which you may want to opt out of). Since one user agent controls both, you can't split the decision — weigh which matters more using our blocking decision guide.
Check and monitor Diffbot on your site
- AI Crawler Access Checker — see whether your current robots.txt allows or blocks Diffbot.
- robots.txt Generator for AI Bots — build a robots.txt with per-bot rules for all 28 known AI crawlers.
- AI Bot Log Analyzer — paste server logs and count real Diffbot hits, entirely in your browser.
Frequently asked questions
What is Diffbot?
Diffbot is Diffbot's crawler serving both AI search features and model training. Extracts structured data from pages for customers, including AI companies.
Does Diffbot respect robots.txt?
Diffbot's bulk crawling honors robots.txt, but customer-triggered, on-demand extraction of specific URLs has been reported to proceed regardless — treat blocks as advisory.
How do I block Diffbot?
Add "User-agent: Diffbot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.
Does blocking Diffbot hurt my Google rankings?
No. Diffbot is separate from Googlebot, which handles Google Search indexing. Blocking Diffbot has no effect on your traditional search rankings, but it does remove your pages from the AI answers Diffbot's assistant serves to its users.
How can I tell if Diffbot is crawling my site?
Search your server access logs for the string "Diffbot". Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Diffbot, with per-path breakdowns.
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-07-18.