Check whether AI crawlers can access your site

How to use the tool

Paste the URL of a page on your site and click “Test”. The tool fetches the robots.txt file at the root of the site, applies its rules to the page’s path and shows, crawler by crawler, whether it is allowed or blocked. The right-hand column quotes the line of the file that decides; those lines are also highlighted in the file shown below the table.

The result applies to the page you entered: a robots.txt can open the blog and close the customer area, so test the pages that matter. To check a change before putting it live, open “Test a robots.txt before publishing it”, paste the new content and a page path: the analysis then runs in your browser, with nothing sent anywhere.

Four families of crawlers

AI companies do not send a single crawler onto the web. Most use several, each with its own name in robots.txt, and each name is controlled separately. OpenAI says so plainly: a site can allow OAI-SearchBot to appear in ChatGPT search results while disallowing GPTBot, so that its content is not used to train the models.¹

• Training. GPTBot, ClaudeBot, Meta-ExternalAgent, MistralAI-Training or Amazonbot collect pages that may be used to train models. Google-Extended and Applebot-Extended work differently: they are control tokens with no crawler of their own, telling Google² and Apple³ whether content already crawled may be used for their models.
• Search and answers. OAI-SearchBot, Claude-SearchBot, PerplexityBot, Meta-WebIndexer, MistralAI-Index, Amzn-SearchBot or DuckAssistBot index pages to cite them in their engine’s answers. This is the family that matters for GEO.
• Visits on user request. ChatGPT-User, Claude-User, Perplexity-User, Meta-ExternalFetcher, Amzn-User or MistralAI-User open a page because a user has just asked a question that needs it.
• Open archive. CCBot feeds Common Crawl, an archive of the web that anyone can download and analyse.

The 23 crawlers tested

The list and the roles come from each company’s documentation, reviewed on October 16, 2026.

Opt out of training without vanishing from answers

This is the most requested setup, and it fits in a few lines. Each training crawler gets its own group, and the “*” group stays open for everyone else:

User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
User-agent: Applebot-Extended
User-agent: Meta-ExternalAgent
User-agent: MistralAI-Training
Disallow: /

User-agent: *
Allow: /

Search crawlers are not named: they follow the “*” group and stay allowed. Closing one of them has effects that the operators describe themselves. According to OpenAI, a site that disallows OAI-SearchBot no longer appears in ChatGPT search answers, except as a navigational link.¹ Anthropic warns that disallowing Claude-SearchBot may reduce the site’s visibility in search results,⁴ and Perplexity recommends allowing PerplexityBot to appear in its results.⁵

Google works differently. Google-Extended has no effect on inclusion in Google Search or on ranking.² AI Overviews and AI Mode are part of Search: Googlebot controls access to them, and Google points to the nosnippet, data-nosnippet, max-snippet and noindex controls to limit what is shown.⁶ Blocking Googlebot to stay out of AI Overviews would mean leaving Google altogether.

How a crawler reads your robots.txt

The tool applies the rules Google describes in its robots.txt specification, which follows the Robots Exclusion Protocol standardised as RFC 9309.¹²

• One group per crawler. The crawler looks for a group with its name. If it finds one, it ignores the “*” group; otherwise the “*” group applies. Several groups with the same name are merged, and consecutive User-agent lines share the same rules.
• The longest rule wins. Between “Allow: /blog/” and “Disallow: /”, the first applies to /blog/article, because it is more specific. When both have the same length, Allow wins.
• Two wildcards. “*” stands for any sequence of characters and “$” marks the end of the URL: “Disallow: /*.pdf$” blocks PDF files, not pages whose URL contains “.pdf?”.
• Case. Field and crawler names are case-insensitive, paths are not: “/Blog/” and “/blog/” are two different rules.
• Missing file or error. A 404 or any other 4xx error except 429 counts as no file: nothing is blocked. A 5xx server error, on the other hand, makes Google pause crawling. Beyond five redirects the file is treated as missing, and only its first 500 KiB are read.¹²

What robots.txt does not guarantee

robots.txt is an instruction, not a lock. Crawlers acting on a user’s request are the first to set it aside, and their operators say so: OpenAI states that robots.txt rules may not apply to ChatGPT-User,¹ Perplexity that Perplexity-User generally ignores them,⁵ Meta that Meta-ExternalFetcher may bypass them,⁷ Amazon that Amzn-User may not follow them all.⁸ The tool flags it when one of them is blocked.

Conversely, a crawler allowed by robots.txt can be turned away earlier, by the host or by a web application firewall. Google reminds site owners that crawling must be allowed in robots.txt and by any CDN or hosting infrastructure,⁶ and Perplexity explains how to open its crawlers in Cloudflare or AWS rules.⁵ Those blocks do not show in robots.txt: only server logs reveal them.

Finally, a crawler’s name can be spoofed. OpenAI, Perplexity, Amazon, Common Crawl and DuckDuckGo publish their crawlers’ IP addresses,^{1, 5, 8, 11, 10} and Apple and Google let you check them with a reverse DNS lookup.^{3, 2} A visit that claims to be GPTBot from another address does not come from OpenAI.

AI crawlers and GEO

A page that search crawlers cannot read will not be cited in ChatGPT, Claude or Perplexity answers. Checking robots.txt is therefore the first step of optimising for generative engines, before any work on the content. On Google’s side, the rules are those of classic search, and the links AI Mode and AI Overviews give to websites depend first on crawling by Googlebot. Once access is confirmed, the next step is to polish what these engines show of your pages, starting with the length of the title and meta description.

Frequently asked questions