GeoPromptTracker

AI crawler profile

What is omgilibot? How to allow or block Webz.io's crawler

User-agent string to match in robots.txt and server logs: omgilibot

Operator

Webz.io

Purpose

AI model training

robots.txt

Respects robots.txt

Official docs

Webz.io documentation ↗

omgilibot (also seen as omgili) crawls for Webz.io, a web-data broker whose feeds — forums, reviews, news, and general web content — are licensed to enterprise customers including AI model builders. Blocking it is less about one AI product and more about opting out of a dataset resold downstream.

Does omgilibot respect robots.txt?

Webz.io documents the bot and states it honors robots.txt; as a data vendor its compliance is part of its sales pitch.

Verify it's really omgilibot

Webz.io publishes no IP-range file for omgilibot, so it can only be identified by its user-agent string — which anything can send. Treat traffic claiming this agent as unverified, and prefer a reverse-DNS check where the operator documents one before acting on it.

Block omgilibot with robots.txt

robots.txt — block
User-agent: omgilibot
Disallow: /

Explicitly allow omgilibot

robots.txt — allow
User-agent: omgilibot
Allow: /

Block omgilibot at the server or CDN

robots.txt is the right first step for omgilibot, since Webz.io honors it. Use these only if you want the block enforced rather than requested — for example to stop agents spoofing the user agent. Matching on the user-agent string still trusts a self-declared header — and no IP-range file exists for this agent, so treat it as best-effort.

nginx
if ($http_user_agent ~* "omgilibot") {
    return 403;
}
apache — .htaccess
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} omgilibot [NC]
RewriteRule .* - [F,L]
cloudflare — WAF expression
(http.user_agent contains "omgilibot")

Find omgilibot in your server logs

shell
grep -i "omgilibot" /var/log/nginx/access.log | wc -l

Should you block omgilibot?

Blocking omgilibot opts your content out of Webz.io's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.

Check and monitor omgilibot on your site

Related reading

Frequently asked questions

What is omgilibot?

omgilibot is Webz.io's web crawler for collecting AI training data. Webz.io's crawler; its web-data feeds are sold to AI builders.

Does omgilibot respect robots.txt?

Webz.io documents the bot and states it honors robots.txt; as a data vendor its compliance is part of its sales pitch.

How do I block omgilibot?

Add "User-agent: omgilibot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.

Does blocking omgilibot hurt my Google rankings?

No. omgilibot is separate from Googlebot, which handles Google Search indexing. Blocking omgilibot has no effect on your traditional search rankings.

How can I tell if omgilibot is crawling my site?

Search your server access logs for the string "omgilibot" — for example: grep -i "omgilibot" /var/log/nginx/access.log | wc -l. Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including omgilibot, with per-path breakdowns.

How do I use omgilibot?

You don't — omgilibot isn't a tool you run. It's Webz.io's own crawler, operated by Webz.io, that visits your site from their infrastructure. The only control you have over it is whether you allow or block it, via robots.txt or a server rule. If you're looking to crawl other sites yourself, you'd write your own crawler or use a crawling library; sending "omgilibot" as your user agent would be impersonating Webz.io.

Where is the official Webz.io documentation for omgilibot?

Webz.io publishes it at https://webz.io/bot.html. That page is the authoritative source for the user-agent string and Webz.io's stated crawling policy.

Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-08-14.