GeoPromptTracker

AI crawler profile

What is Diffbot? How to allow or block Diffbot's crawler

User-agent string to match in robots.txt and server logs: Diffbot

Operator

Diffbot

Purpose

Training + search

robots.txt

Partially respects robots.txt

Official docs

Diffbot documentation ↗

Diffbot converts web pages into structured data (its Knowledge Graph) that customers — including AI companies — query and license. It's less a classic crawler than an extraction service: many fetches happen on demand when a customer requests a specific URL, which complicates the usual robots.txt picture.

Does Diffbot respect robots.txt?

Diffbot's bulk crawling honors robots.txt, but customer-triggered, on-demand extraction of specific URLs has been reported to proceed regardless — treat blocks as advisory.

Verify it's really Diffbot

Diffbot publishes no IP-range file for Diffbot, so it can only be identified by its user-agent string — which anything can send. Treat traffic claiming this agent as unverified, and prefer a reverse-DNS check where the operator documents one before acting on it.

Block Diffbot with robots.txt

robots.txt — block
User-agent: Diffbot
Disallow: /

Explicitly allow Diffbot

robots.txt — allow
User-agent: Diffbot
Allow: /

Block Diffbot at the server or CDN

Diffbot's robots.txt compliance is unreliable, so a robots.txt rule alone may not stop it. These enforce the block. Matching on the user-agent string still trusts a self-declared header — and no IP-range file exists for this agent, so treat it as best-effort.

nginx
if ($http_user_agent ~* "Diffbot") {
    return 403;
}
apache — .htaccess
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Diffbot [NC]
RewriteRule .* - [F,L]
cloudflare — WAF expression
(http.user_agent contains "Diffbot")

Find Diffbot in your server logs

shell
grep -i "Diffbot" /var/log/nginx/access.log | wc -l

Should you block Diffbot?

It's a genuine trade-off: Diffbotserves both search-style features (which send you visibility) and model training (which you may want to opt out of). Since one user agent controls both, you can't split the decision — weigh which matters more using our blocking decision guide.

Check and monitor Diffbot on your site

Related reading

Frequently asked questions

What is Diffbot?

Diffbot is Diffbot's crawler serving both AI search features and model training. Extracts structured data from pages for customers, including AI companies.

Does Diffbot respect robots.txt?

Diffbot's bulk crawling honors robots.txt, but customer-triggered, on-demand extraction of specific URLs has been reported to proceed regardless — treat blocks as advisory.

How do I block Diffbot?

Add "User-agent: Diffbot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.

Does blocking Diffbot hurt my Google rankings?

No. Diffbot is separate from Googlebot, which handles Google Search indexing. Blocking Diffbot has no effect on your traditional search rankings, but it does remove your pages from the AI answers Diffbot's assistant serves to its users.

How can I tell if Diffbot is crawling my site?

Search your server access logs for the string "Diffbot" — for example: grep -i "Diffbot" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Diffbot, with per-path breakdowns.

How do I use Diffbot?

You don't — Diffbot isn't a tool you run. It's Diffbot's own crawler, operated by Diffbot, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "Diffbot" as your user agent would be impersonating Diffbot.

Where is the official Diffbot documentation for Diffbot?

Diffbot publishes it at https://docs.diffbot.com/docs/how-to-identify-diffbot. That page is the authoritative source for the user-agent string and Diffbot's stated crawling policy.

Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-08-14.