Our bot
webAInalizer-Bot
If you saw "webAInalizer-Bot" in your server logs, it is our analyzer. It only reads public pages, and only when someone requests an analysis.
When it visits a site
When someone analyzes their site on webainalizer.com, and in the full report, when an AI engine mentions a competitor of that site: we then also read that competitor's public pages to compare them. We do not crawl the web on our own.
What it reads and how fast
robots.txt, llms.txt, the sitemap, the home page and at most 10 pages, one at a time, with at least 400 ms between requests and a 90-second limit per site. It does not submit forms, log in or store anything that is not public.
How to block it
Add to your robots.txt: "User-agent: webainalizer-bot" and on the next line "Disallow: /". With that we will not visit your site, not even the home page. We also respect inner pages you block for all bots; in that case we still read the home page, so we can tell the owner if their robots.txt blocks everything by mistake.
Frequently asked questions
How do I recognize it?
Its user agent starts with "webAInalizer-Bot/" and links to this page.
Who do I contact if it causes a problem?
Write to [email protected] with your domain and the approximate time.