Loading
AI crawler profile
User-agent string to match in robots.txt and server logs: PanguBot
Operator
Huawei
Purpose
AI model training
robots.txt
Compliance unknown
Official docs
None published
PanguBot is the user agent linked to data collection for Huawei's PanGu family of large language models. Unlike PetalBot (Huawei's documented search crawler), PanguBot has no official documentation, putting it in the same category as Bytespider: a training crawler you can only reliably control at the firewall level.
No official documentation exists and independent compliance reports are limited — pair any robots.txt block with a firewall or CDN rule if opting out matters to you.
Huawei publishes no IP-range file for PanguBot, so it can only be identified by its user-agent string — which anything can send. Treat traffic claiming this agent as unverified, and prefer a reverse-DNS check where the operator documents one before acting on it.
User-agent: PanguBot Disallow: /
⚠ Because PanguBot's robots.txt compliance is unreliable, pair this with a firewall or CDN rule matching the user-agent string if blocking actually matters to you.
User-agent: PanguBot Allow: /
PanguBot's robots.txt compliance is unreliable, so a robots.txt rule alone may not stop it. These enforce the block. Matching on the user-agent string still trusts a self-declared header — and no IP-range file exists for this agent, so treat it as best-effort.
if ($http_user_agent ~* "PanguBot") {
return 403;
}RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} PanguBot [NC]
RewriteRule .* - [F,L](http.user_agent contains "PanguBot")
grep -i "PanguBot" /var/log/nginx/access.log | wc -l
Blocking PanguBot opts your content out of Huawei's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.
PanguBot is Huawei's web crawler for collecting AI training data. Associated with training Huawei's PanGu large language models.
No official documentation exists and independent compliance reports are limited — pair any robots.txt block with a firewall or CDN rule if opting out matters to you.
Add "User-agent: PanguBot" followed by "Disallow: /" to your robots.txt file. Because this bot's robots.txt compliance is not reliable, enforce the block with firewall or CDN rules (for example a Cloudflare WAF rule matching the user agent) if it matters to you.
No. PanguBot is separate from Googlebot, which handles Google Search indexing. Blocking PanguBot has no effect on your traditional search rankings.
Search your server access logs for the string "PanguBot" — for example: grep -i "PanguBot" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including PanguBot, with per-path breakdowns.
You don't — PanguBot isn't a tool you run. It's Huawei's own crawler, operated by Huawei, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "PanguBot" as your user agent would be impersonating Huawei.
Almost certainly not. Crawl-delay was never part of the original robots.txt specification — Google has publicly said it ignores the directive, and the AI crawlers that model their parsers on Google's do the same. Huawei publishes no position on crawl-delay for PanguBot, so treat it as unsupported. If PanguBot is hitting your site harder than you want, rate-limit it at the server or CDN instead: a Cloudflare rate-limiting rule or an nginx limit_req zone matched on the user agent will actually be enforced, whereas a crawl-delay line is only a request that this agent likely never reads.
Huawei makes no enforceable commitment that PanguBot honors robots.txt, so the plain answer may be that it simply does not. Before assuming that, rule out the ordinary causes: a cached copy of robots.txt from before your change, a user-agent line that does not actually match, or a block placed in a group this agent does not read. If the hits continue after those are excluded, robots.txt is not going to stop this agent and you need a firewall or CDN rule that returns 403.
Treat it as no unless Huawei says otherwise. Rendering JavaScript costs an order of magnitude more than fetching HTML, and most AI crawlers — unlike Googlebot, which runs a full headless Chrome — read the raw HTML response and stop there. The practical consequence: anything your page loads client-side after the initial response is likely invisible to PanguBot. If your main content is client-rendered, server-render it or pre-render it so the text exists in the first response.
Huawei publishes no IP-range file for PanguBot, which means there is no way to verify a request really came from them. Any traffic claiming this user agent should be treated as unverified. If you need certainty, block on the user agent and accept that you may be blocking impostors rather than the real crawler — and note that the absence of published ranges is itself a signal about how seriously the operator treats crawler transparency.
There is no published schedule, and no operator commits to one. Training crawlers like PanguBot typically sweep in bursts rather than at a steady rate — quiet for weeks, then hundreds of requests over a day or two as a collection run reaches your domain. Larger and more-linked sites are revisited more often. The only way to know for your own site is to measure it: grep your access log for the user agent, or paste the log into our AI Bot Log Analyzer, which breaks hits down by date and path in your browser.
Huawei publishes no official documentation for PanguBot. That absence is itself worth noting: an undocumented crawler gives you no stated policy to hold it to, and no published IP ranges to verify it against.
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-09-13.