GeoPromptTracker

AI crawler profile

What is AI2Bot? How to allow or block Allen Institute for AI's crawler

User-agent string to match in robots.txt and server logs: AI2Bot

Operator

Allen Institute for AI

Purpose

AI model training

robots.txt

Respects robots.txt

Official docs

Allen Institute for AI documentation ↗

AI2Bot collects web content for the Allen Institute for AI (Ai2), the nonprofit behind open models like OLMo and open datasets like Dolma. Content it gathers can end up in openly released research corpora — meaning a block here is a decision about open research datasets, not just one company's product.

Does AI2Bot respect robots.txt?

Ai2 documents the crawler and honors robots.txt; as a research nonprofit it has strong incentives to stay compliant.

Verify it's really AI2Bot

Allen Institute for AI publishes no IP-range file for AI2Bot, so it can only be identified by its user-agent string — which anything can send. Treat traffic claiming this agent as unverified, and prefer a reverse-DNS check where the operator documents one before acting on it.

Block AI2Bot with robots.txt

robots.txt — block
User-agent: AI2Bot
Disallow: /

Explicitly allow AI2Bot

robots.txt — allow
User-agent: AI2Bot
Allow: /

Block AI2Bot at the server or CDN

robots.txt is the right first step for AI2Bot, since Allen Institute for AI honors it. Use these only if you want the block enforced rather than requested — for example to stop agents spoofing the user agent. Matching on the user-agent string still trusts a self-declared header — and no IP-range file exists for this agent, so treat it as best-effort.

nginx
if ($http_user_agent ~* "AI2Bot") {
    return 403;
}
apache — .htaccess
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AI2Bot [NC]
RewriteRule .* - [F,L]
cloudflare — WAF expression
(http.user_agent contains "AI2Bot")

Find AI2Bot in your server logs

shell
grep -i "AI2Bot" /var/log/nginx/access.log | wc -l

Should you block AI2Bot?

Blocking AI2Bot opts your content out of Allen Institute for AI's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.

Check and monitor AI2Bot on your site

Related reading

Frequently asked questions

What is AI2Bot?

AI2Bot is Allen Institute for AI's web crawler for collecting AI training data. Crawls the web for the Allen Institute's open AI research models.

Does AI2Bot respect robots.txt?

Ai2 documents the crawler and honors robots.txt; as a research nonprofit it has strong incentives to stay compliant.

How do I block AI2Bot?

Add "User-agent: AI2Bot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.

Does blocking AI2Bot hurt my Google rankings?

No. AI2Bot is separate from Googlebot, which handles Google Search indexing. Blocking AI2Bot has no effect on your traditional search rankings.

How can I tell if AI2Bot is crawling my site?

Search your server access logs for the string "AI2Bot" — for example: grep -i "AI2Bot" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including AI2Bot, with per-path breakdowns.

How do I use AI2Bot?

You don't — AI2Bot isn't a tool you run. It's Allen Institute for AI's own crawler, operated by Allen Institute for AI, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "AI2Bot" as your user agent would be impersonating Allen Institute for AI.

Where is the official Allen Institute for AI documentation for AI2Bot?

Allen Institute for AI publishes it at https://allenai.org/crawler. That page is the authoritative source for the user-agent string and Allen Institute for AI's stated crawling policy.

Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-08-14.