AI crawler profile
What is AI2Bot? How to allow or block Allen Institute for AI's crawler
User-agent string to match in robots.txt and server logs: AI2Bot
Operator
Allen Institute for AI
Purpose
AI model training
robots.txt
Respects robots.txt
Official docs
AI2Bot collects web content for the Allen Institute for AI (Ai2), the nonprofit behind open models like OLMo and open datasets like Dolma. Content it gathers can end up in openly released research corpora — meaning a block here is a decision about open research datasets, not just one company's product.
Does AI2Bot respect robots.txt?
Ai2 documents the crawler and honors robots.txt; as a research nonprofit it has strong incentives to stay compliant.
Verify it's really AI2Bot
Allen Institute for AI publishes no IP-range file for AI2Bot, so it can only be identified by its user-agent string — which anything can send. Treat traffic claiming this agent as unverified, and prefer a reverse-DNS check where the operator documents one before acting on it.
Block AI2Bot with robots.txt
User-agent: AI2Bot Disallow: /
Explicitly allow AI2Bot
User-agent: AI2Bot Allow: /
Block AI2Bot at the server or CDN
robots.txt is the right first step for AI2Bot, since Allen Institute for AI honors it. Use these only if you want the block enforced rather than requested — for example to stop agents spoofing the user agent. Matching on the user-agent string still trusts a self-declared header — and no IP-range file exists for this agent, so treat it as best-effort.
if ($http_user_agent ~* "AI2Bot") {
return 403;
}RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} AI2Bot [NC]
RewriteRule .* - [F,L](http.user_agent contains "AI2Bot")
Find AI2Bot in your server logs
grep -i "AI2Bot" /var/log/nginx/access.log | wc -l
Should you block AI2Bot?
Blocking AI2Bot opts your content out of Allen Institute for AI's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.
Check and monitor AI2Bot on your site
- AI Crawler Access Checker — see whether your current robots.txt allows or blocks AI2Bot.
- robots.txt Generator for AI Bots — build a robots.txt with per-bot rules for all 28 known AI crawlers.
- AI Bot Log Analyzer — paste server logs and count real AI2Bot hits, entirely in your browser.
Related reading
- List of AI crawlers and their user agents — how AI2Bot fits among every other AI crawler, in one table.
- Can AI crawlers execute JavaScript? — whether AI2Bot sees content your site renders client-side.
- Should you block AI bots? — the full decision framework for Allen Institute for AI and the rest.
Frequently asked questions
What is AI2Bot?
AI2Bot is Allen Institute for AI's web crawler for collecting AI training data. Crawls the web for the Allen Institute's open AI research models.
Does AI2Bot respect robots.txt?
Ai2 documents the crawler and honors robots.txt; as a research nonprofit it has strong incentives to stay compliant.
How do I block AI2Bot?
Add "User-agent: AI2Bot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.
Does blocking AI2Bot hurt my Google rankings?
No. AI2Bot is separate from Googlebot, which handles Google Search indexing. Blocking AI2Bot has no effect on your traditional search rankings.
How can I tell if AI2Bot is crawling my site?
Search your server access logs for the string "AI2Bot" — for example: grep -i "AI2Bot" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including AI2Bot, with per-path breakdowns.
How do I use AI2Bot?
You don't — AI2Bot isn't a tool you run. It's Allen Institute for AI's own crawler, operated by Allen Institute for AI, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "AI2Bot" as your user agent would be impersonating Allen Institute for AI.
Does AI2Bot respect crawl-delay?
Almost certainly not. Crawl-delay was never part of the original robots.txt specification — Google has publicly said it ignores the directive, and the AI crawlers that model their parsers on Google's do the same. Allen Institute for AI publishes no position on crawl-delay for AI2Bot, so treat it as unsupported. If AI2Bot is hitting your site harder than you want, rate-limit it at the server or CDN instead: a Cloudflare rate-limiting rule or an nginx limit_req zone matched on the user agent will actually be enforced, whereas a crawl-delay line is only a request that this agent likely never reads.
Why is AI2Bot ignoring my robots.txt?
Allen Institute for AI states AI2Bot honors robots.txt, so if you are still seeing hits the cause is usually one of four things rather than the bot misbehaving. First, robots.txt is cached — Allen Institute for AI may be working from a copy fetched up to 24 hours before your change. Second, the rule may not match: robots.txt user-agent matching is on a prefix of the token, and a typo or a trailing character breaks it silently. Third, the block may sit under a different user-agent group than the one this bot reads, since a bot obeys only the most specific group that matches it, not the wildcard group as well. Fourth, the traffic may be something else sending AI2Bot as its user agent, which anything can do.
Does AI2Bot execute JavaScript?
Treat it as no unless Allen Institute for AI says otherwise. Rendering JavaScript costs an order of magnitude more than fetching HTML, and most AI crawlers — unlike Googlebot, which runs a full headless Chrome — read the raw HTML response and stop there. The practical consequence: anything your page loads client-side after the initial response is likely invisible to AI2Bot. If your main content is client-rendered, server-render it or pre-render it so the text exists in the first response.
What are AI2Bot's IP ranges?
Allen Institute for AI publishes no IP-range file for AI2Bot, which means there is no way to verify a request really came from them. Any traffic claiming this user agent should be treated as unverified. If you need certainty, block on the user agent and accept that you may be blocking impostors rather than the real crawler — and note that the absence of published ranges is itself a signal about how seriously the operator treats crawler transparency.
How often does AI2Bot crawl my site?
There is no published schedule, and no operator commits to one. Training crawlers like AI2Bot typically sweep in bursts rather than at a steady rate — quiet for weeks, then hundreds of requests over a day or two as a collection run reaches your domain. Larger and more-linked sites are revisited more often. The only way to know for your own site is to measure it: grep your access log for the user agent, or paste the log into our AI Bot Log Analyzer, which breaks hits down by date and path in your browser.
Where is the official Allen Institute for AI documentation for AI2Bot?
Allen Institute for AI publishes it at https://allenai.org/crawler. That page is the authoritative source for the user-agent string and Allen Institute for AI's stated crawling policy.
Other AI crawlers
- ImagesiftBotCrawls images across the web for Hive's visual-AI products.
- omgilibotWebz.io's crawler; its web-data feeds are sold to AI builders.
- PanguBotAssociated with training Huawei's PanGu large language models.
- GPTBotCrawls content to train OpenAI's models.
- ClaudeBotCrawls content to train Anthropic's Claude models.
- Google-ExtendedControls use of content for Gemini and Vertex AI training, separate from Googlebot.
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-09-13.