GeoPromptTracker

AI crawler profile

What is GPTBot? How to allow or block OpenAI's crawler

User-agent string to match in robots.txt and server logs: GPTBot

Operator

OpenAI

Purpose

AI model training

robots.txt

Respects robots.txt

Official docs

OpenAI documentation ↗

GPTBot is OpenAI's web crawler for gathering training data. Content it collects may be used to improve future GPT models. It is OpenAI's broadest crawler, and the one most site owners mean when they talk about "blocking ChatGPT" — though blocking it does not remove you from ChatGPT's live search answers, which use OAI-SearchBot instead.

Does GPTBot respect robots.txt?

OpenAI documents GPTBot's IP ranges and states it honors robots.txt disallow rules.

Verify it's really GPTBot

The user-agent string above is self-declared, so anything can send it. OpenAI publishes GPTBot's IP ranges as JSON, which is what makes a rule verifiable: check the request's IP against the published prefixes instead of trusting the name. Requests claiming to be GPTBot from outside those ranges are spoofed — useful to know whether you're allowing or blocking it.

Block GPTBot with robots.txt

robots.txt — block
User-agent: GPTBot
Disallow: /

Explicitly allow GPTBot

robots.txt — allow
User-agent: GPTBot
Allow: /

Block GPTBot at the server or CDN

robots.txt is the right first step for GPTBot, since OpenAI honors it. Use these only if you want the block enforced rather than requested — for example to stop agents spoofing the user agent. Matching on the user-agent string still trusts a self-declared header — pair it with an IP check against the published ranges above for a rule that can't be spoofed.

nginx
if ($http_user_agent ~* "GPTBot") {
    return 403;
}
apache — .htaccess
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} GPTBot [NC]
RewriteRule .* - [F,L]
cloudflare — WAF expression
(http.user_agent contains "GPTBot")

Find GPTBot in your server logs

shell
grep -i "GPTBot" /var/log/nginx/access.log | wc -l

Should you block GPTBot?

Blocking GPTBot opts your content out of OpenAI's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.

Check and monitor GPTBot on your site

Related reading

Frequently asked questions

What is GPTBot?

GPTBot is OpenAI's web crawler for collecting AI training data. Crawls content to train OpenAI's models.

Does GPTBot respect robots.txt?

OpenAI documents GPTBot's IP ranges and states it honors robots.txt disallow rules.

How do I block GPTBot?

Add "User-agent: GPTBot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.

Does blocking GPTBot hurt my Google rankings?

No. GPTBot is separate from Googlebot, which handles Google Search indexing. Blocking GPTBot has no effect on your traditional search rankings.

How can I tell if GPTBot is crawling my site?

Search your server access logs for the string "GPTBot" — for example: grep -i "GPTBot" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including GPTBot, with per-path breakdowns.

How do I use GPTBot?

You don't — GPTBot isn't a tool you run. It's OpenAI's own crawler, operated by OpenAI, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "GPTBot" as your user agent would be impersonating OpenAI.

Does GPTBot respect crawl-delay?

Almost certainly not. Crawl-delay was never part of the original robots.txt specification — Google has publicly said it ignores the directive, and the AI crawlers that model their parsers on Google's do the same. OpenAI publishes no position on crawl-delay for GPTBot, so treat it as unsupported. If GPTBot is hitting your site harder than you want, rate-limit it at the server or CDN instead: a Cloudflare rate-limiting rule or an nginx limit_req zone matched on the user agent will actually be enforced, whereas a crawl-delay line is only a request that this agent likely never reads.

Why is GPTBot ignoring my robots.txt?

OpenAI states GPTBot honors robots.txt, so if you are still seeing hits the cause is usually one of four things rather than the bot misbehaving. First, robots.txt is cached — OpenAI may be working from a copy fetched up to 24 hours before your change. Second, the rule may not match: robots.txt user-agent matching is on a prefix of the token, and a typo or a trailing character breaks it silently. Third, the block may sit under a different user-agent group than the one this bot reads, since a bot obeys only the most specific group that matches it, not the wildcard group as well. Fourth, the traffic may be something else sending GPTBot as its user agent, which anything can do.

Does GPTBot execute JavaScript?

Treat it as no unless OpenAI says otherwise. Rendering JavaScript costs an order of magnitude more than fetching HTML, and most AI crawlers — unlike Googlebot, which runs a full headless Chrome — read the raw HTML response and stop there. The practical consequence: anything your page loads client-side after the initial response is likely invisible to GPTBot. If your main content is client-rendered, server-render it or pre-render it so the text exists in the first response.

What are GPTBot's IP ranges?

OpenAI publishes them as JSON at https://openai.com/gptbot.json, updated as their infrastructure changes. Fetch that file rather than hard-coding the prefixes, and match the request IP against it to confirm a visit is genuinely GPTBot. This is the only reliable check — the user-agent header is self-declared, so a scraper can send "GPTBot" while coming from anywhere.

How often does GPTBot crawl my site?

There is no published schedule, and no operator commits to one. Training crawlers like GPTBot typically sweep in bursts rather than at a steady rate — quiet for weeks, then hundreds of requests over a day or two as a collection run reaches your domain. Larger and more-linked sites are revisited more often. The only way to know for your own site is to measure it: grep your access log for the user agent, or paste the log into our AI Bot Log Analyzer, which breaks hits down by date and path in your browser.

Where is the official OpenAI documentation for GPTBot?

OpenAI publishes it at https://platform.openai.com/docs/bots. That page is the authoritative source for the user-agent string and OpenAI's stated crawling policy, and OpenAI also publishes GPTBot's IP ranges as JSON at https://openai.com/gptbot.json so you can verify requests rather than trusting the header.

Commonly confused with

Other AI crawlers

Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-09-13.