Test real bot access to a page

How to use the tool

Paste the address of a page on your site and click “Test”. The nemociment.fr server requests the page posing as a Chrome browser, then as each of the eight bots, and the tool compares every answer with the browser’s. A bot that gets a different status, a challenge page or other content is flagged in red or orange, with the reason.

Test the home page, but also an article and a service page: firewall rules sometimes target specific paths. The test takes 5 to 15 seconds depending on the speed of the site.

What the tool compares

Each request follows redirects by hand, five at most, and every step is rechecked before it is followed. Of the page body, the tool keeps only its size and its title, and discards the rest. It then compares each bot with the reference browser:

• Refusal: 401, 403, 429 or 451 codes, and 5xx server errors the browser does not get.
• Challenge page: marks of Cloudflare, Anubis, Akamai, Imperva, DataDome, Sucuri and AWS WAF in the page or its headers.
• Different redirect: the bot does not end up at the same address as the browser, or takes another path.
• Very different content: half or double the size, or a different title, on two 200 responses. The size comparison is skipped when one page exceeds the 256 KiB reading limit.

The user-agents sent

Each bot is announced by the name its publisher documents. The table gives each one’s role according to the publisher and the source that describes it.

Two notes on the strings sent. The reference Chrome and the Chrome version in the Googlebot smartphone string are stand-in values: Google publishes a generic “W.X.Y.Z” number that its crawler replaces with the Chromium version it uses.⁴ As for Anthropic, its help article names the three bots without giving their full string:² the tool therefore sends the documented name inside a string shaped like the other publishers’, which is enough for a firewall rule that looks for that name.

Google-Extended is not tested. Google states that it has no separate user-agent string: crawling goes through Google’s existing user-agents, and Google-Extended is only a control name in robots.txt.⁴ There is therefore no request to attribute to it.

What the tool cannot know

The request leaves from the nemociment.fr server, not from the real bots’ IP addresses. A firewall can check where a visit comes from: Cloudflare treats a bot as “verified” when its identity can be proven by a cryptographic signature, by a published IP list with a stable user-agent, or by reverse DNS,⁵ and Perplexity advises writing its Cloudflare rule by combining the bot’s name with its published IP ranges.³ A request that carries the right name but comes from another address does not match such a rule: it may be treated differently from the real visit.

So the two readings of the result are not equal. An OK does not prove that the real bots get through: the firewall may let a fake name in and keep different treatment for the real addresses, or the opposite. A block only proves that a visitor presenting itself this way is blocked, which already points to a rule on the user-agent. To see what really happens, look at the real bots’ visits in your server logs with the log analyzer, and check that an address really belongs to the publisher with the bot IP checker.

The tool does not read robots.txt either: that is the job of the AI crawler access test. A bot can be allowed by robots.txt and refused by the firewall, or the other way round, and the two tools complement each other.

Fixing a block

If a bot you want to welcome is refused, the cause is almost always a rule at the host, a CDN or a security plugin. On Cloudflare, AI bot protection sorts bots into three separately adjustable families, search, agent and training: each blocking option targets the verified bots of the family and also unverified bots that behave the same way.⁶ According to the same page, new domains get from September 15, 2026 a default setting that blocks training and agent bots on pages with ads and leaves search open.⁶

To open a specific bot, Perplexity describes an allow rule in Cloudflare or AWS WAF that pairs the bot’s name with its published IP addresses, placed before the blocking rules, with the ranges refreshed regularly.³ After any change, run the test again and watch the logs: a rule can take time to propagate.

Going further

This test sits between robots.txt and the logs. For the full method, read the article on generative engine optimization, and look at the page’s HTTP headers when an unexpected status shows up.

Frequently asked questions