AI crawler profile
What is omgilibot? How to allow or block Webz.io's crawler
Operator
Webz.io
Purpose
AI model training
robots.txt
Respects robots.txt
Official docs
omgilibot (also seen as omgili) crawls for Webz.io, a web-data broker whose feeds — forums, reviews, news, and general web content — are licensed to enterprise customers including AI model builders. Blocking it is less about one AI product and more about opting out of a dataset resold downstream.
Does omgilibot respect robots.txt?
Webz.io documents the bot and states it honors robots.txt; as a data vendor its compliance is part of its sales pitch.
Block omgilibot with robots.txt
User-agent: omgilibot Disallow: /
Explicitly allow omgilibot
User-agent: omgilibot Allow: /
Should you block omgilibot?
Blocking omgilibot opts your content out of Webz.io's model training — a legitimate choice for original content you don't want reproduced by AI. The trade-off: models trained without your content are less likely to know your brand or recommend it unprompted. There is no direct traffic loss today, since training crawlers don't send visitors. See our guide on whether to block AI bots for the full decision framework.
Check and monitor omgilibot on your site
- AI Crawler Access Checker — see whether your current robots.txt allows or blocks omgilibot.
- robots.txt Generator for AI Bots — build a robots.txt with per-bot rules for all 28 known AI crawlers.
- AI Bot Log Analyzer — paste server logs and count real omgilibot hits, entirely in your browser.
Frequently asked questions
What is omgilibot?
omgilibot is Webz.io's web crawler for collecting AI training data. Webz.io's crawler; its web-data feeds are sold to AI builders.
Does omgilibot respect robots.txt?
Webz.io documents the bot and states it honors robots.txt; as a data vendor its compliance is part of its sales pitch.
How do I block omgilibot?
Add "User-agent: omgilibot" followed by "Disallow: /" to your robots.txt file. The change takes effect the next time the bot fetches your robots.txt.
Does blocking omgilibot hurt my Google rankings?
No. omgilibot is separate from Googlebot, which handles Google Search indexing. Blocking omgilibot has no effect on your traditional search rankings.
How can I tell if omgilibot is crawling my site?
Search your server access logs for the string "omgilibot". Our free AI Bot Log Analyzer does this in your browser: paste a log file and it counts hits per AI crawler, including omgilibot, with per-path breakdowns.
Part of our directory of every known AI crawler, refreshed monthly. Last verified: 2026-07-18.