GeoPromptTracker

AI crawler profile

What is Bytespider? How to allow or block ByteDance's crawler

User-agent string to match in robots.txt and server logs: Bytespider

Operator

ByteDance

Purpose

AI model training

robots.txt

Ignores robots.txt

Official docs

None published

Bytespider is ByteDance's aggressive web crawler, believed to gather training data for its AI products including the Doubao models. It has drawn criticism for crawling at very high volume and for continuing to fetch pages on sites that have disallowed it.

Does Bytespider respect robots.txt?

ByteDance publishes no crawler documentation, and multiple independent reports have found Bytespider ignoring robots.txt; blocking it reliably requires firewall or CDN rules.

Verify it's really Bytespider

ByteDance publishes no IP-range file for Bytespider, so it can only be identified by its user-agent string — which anything can send. Treat traffic claiming this agent as unverified, and prefer a reverse-DNS check where the operator documents one before acting on it.

Block Bytespider with robots.txt

robots.txt — block
User-agent: Bytespider
Disallow: /

⚠ Because Bytespider's robots.txt compliance is unreliable, pair this with a firewall or CDN rule matching the user-agent string if blocking actually matters to you.

Explicitly allow Bytespider

robots.txt — allow
User-agent: Bytespider
Allow: /

Block Bytespider at the server or CDN

Bytespider's robots.txt compliance is absent, so a robots.txt rule alone may not stop it. These enforce the block. Matching on the user-agent string still trusts a self-declared header — and no IP-range file exists for this agent, so treat it as best-effort.

nginx
if ($http_user_agent ~* "Bytespider") {
    return 403;
}
apache — .htaccess
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} Bytespider [NC]
RewriteRule .* - [F,L]
cloudflare — WAF expression
(http.user_agent contains "Bytespider")

Find Bytespider in your server logs

shell
grep -i "Bytespider" /var/log/nginx/access.log | wc -l

Should you block Bytespider?

Blocking Bytespider opts your content out of ByteDance's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.

Check and monitor Bytespider on your site

Related reading

Frequently asked questions

What is Bytespider?

Bytespider is ByteDance's web crawler for collecting AI training data. ByteDance's crawler, used for training its AI models.

Does Bytespider respect robots.txt?

ByteDance publishes no crawler documentation, and multiple independent reports have found Bytespider ignoring robots.txt; blocking it reliably requires firewall or CDN rules.

How do I block Bytespider?

Add "User-agent: Bytespider" followed by "Disallow: /" to your robots.txt file. Because this bot's robots.txt compliance is not reliable, enforce the block with firewall or CDN rules (for example a Cloudflare WAF rule matching the user agent) if it matters to you.

Does blocking Bytespider hurt my Google rankings?

No. Bytespider is separate from Googlebot, which handles Google Search indexing. Blocking Bytespider has no effect on your traditional search rankings.

How can I tell if Bytespider is crawling my site?

Search your server access logs for the string "Bytespider" — for example: grep -i "Bytespider" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including Bytespider, with per-path breakdowns.

How do I use Bytespider?

You don't — Bytespider isn't a tool you run. It's ByteDance's own crawler, operated by ByteDance, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "Bytespider" as your user agent would be impersonating ByteDance.

Where is the official ByteDance documentation for Bytespider?

ByteDance publishes no official documentation for Bytespider. That absence is itself worth noting: an undocumented crawler gives you no stated policy to hold it to, and no published IP ranges to verify it against.

Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-08-14.