robots.txt generator and URL tester

How to use the tool

The generator fills in as you choose. Say whether the whole site stays open, add the folders or URLs to close, tick the AI crawlers you do not want and give your sitemap URL. The file is built below, with warnings when a choice may cost you visibility: a fully blocked site, a badly written path, AI search crawlers excluded. Copy or download it, then upload it to the root of your site.

The tester answers one specific question: is this robot allowed to crawl this URL? Paste a full URL so it reads the site’s robots.txt, or choose to test the file you just generated or pasted. It shows the verdict, the group of rules used, the deciding line and the lines Google ignores, with the numbered file below. A file read online is also copied into the text box: choose “The file below” to edit it and run the test again before publishing a change.

What a robots.txt file is for

A robots.txt file tells crawlers which URLs they can access on your site. Its main purpose is to avoid overloading the site with requests.¹ It also keeps pages of no search value out of the crawl: internal search results, carts, endless filtered listings.

It is not a way to keep a page out of Google. A blocked URL can still be indexed if other pages link to it; it then appears without a description. To remove a page from the results, Google recommends a noindex rule, password protection or deleting the page. And for noindex to work, the page must not be blocked in robots.txt, otherwise the crawler never sees the rule.²

The file remains a request: Googlebot and reputable crawlers follow it, others may ignore it. Anything that must stay private needs server-side protection.

Where to put the file and how to write it

The basic rules come from Google’s documentation on creating a robots.txt file:³

• the file is named robots.txt and sits at the root of the site, for example https://www.example.com/robots.txt; a site has only one;
• it only applies to the protocol, host and port where it is published: the one on https://example.com applies neither to https://m.example.com nor to http://example.com;
• it is a UTF-8 text file, best written in a plain text editor rather than a word processor, which adds curly quotes and other stray characters;
• Google reads the first 500 kibibytes and ignores the rest.⁴

The file is made of groups. Each group starts with one or more User-agent lines naming the robots concerned, followed by Disallow and Allow rules. Paths start with “/” and are case-sensitive: /Admin/ and /admin/ are two different paths. Sitemap lines can go anywhere and give the full URL of a sitemap, which helps robots find your pages, as explained in our piece on what Google says about XML sitemaps.

User-agent: *
Disallow: /cart/
Disallow: /*?sort=
Allow: /cart/help/

User-agent: GPTBot
User-agent: CCBot
Disallow: /

Sitemap: https://www.example.com/sitemap.xml

In this example, every robot can crawl the site except the cart, whose help page stays open, and sorted listings. GPTBot and CCBot, gathered in one group, are kept out of the whole site.

How a robot picks the rule that applies

This is the part the tool works out for you, following Google’s specification:⁴

• One group per robot. A robot uses the group that carries its name. If it finds several, it merges them. If it finds none, it follows the “*” group. It never applies both: a robot named in its own group ignores the “*” rules.
• The Googlebot family. Googlebot-Image or Googlebot-Video follow the Googlebot group when no group carries their exact name.
• The longest rule wins. Between “Disallow: /” and “Allow: /blog/”, the second applies to /blog/post. At equal length, Google picks the least restrictive rule, so Allow.
• Two special characters. “*” stands for any sequence of characters; “$” at the end means the URL must stop there. “Disallow: /*.pdf$” blocks every PDF, and only them.
• An empty rule blocks nothing. “Disallow:” with no path allows everything.

What Google ignores

Google only supports four fields: user-agent, allow, disallow and sitemap. Crawl-delay, though widespread, is not supported;⁴ other robots, such as ClaudeBot according to Anthropic, do follow it.⁵ Google also ignores malformed lines and rules placed before the first User-agent line; the tester lists them so you can fix them. Finally, Google generally caches the file for 24 hours: a change does not take effect instantly.⁴

When the file cannot be read, everything depends on the server response. A 4xx error, 404 first among them, means to Google that there is no file: no restrictions. A 5xx error makes it pause crawling the site and retry. Beyond five redirects, the file is treated as missing.⁴

Blocking AI crawlers

AI companies each publish their crawlers, and most of them separate several by purpose. OpenAI separates GPTBot, which collects content to train its models, from OAI-SearchBot, which surfaces sites in ChatGPT search; the two settings are independent.⁶ Blocking GPTBot therefore does not stop you being cited in ChatGPT, whereas blocking OAI-SearchBot removes the site from its search answers.

At Google, Google-Extended controls whether content is used to train Gemini and ground its answers; it has no effect on Google Search. AI Overviews and AI Mode depend on Googlebot, like the rest of Search.⁷ The generator therefore sorts AI crawlers into three families: training, search and answers, visits on user request. The latter act when a user asks the AI to look at a page, and several companies, including OpenAI and Perplexity, say they may not apply robots.txt in that case.^{6, 8} This choice has a direct effect on your visibility in generative engines.

The most common mistakes

• The “Disallow: /” left behind after launch. Handy on a staging site, it blocks the whole site once published. It is one of the first things to check during a website redesign.
• Blocking a page to remove it from Google. It can stay indexed, without a description; and if it carries a noindex, Google will not see it.
• Blocking CSS and JavaScript files. Google advises against blocking resources it needs to understand a page.¹
• Forgetting that a named group replaces “*”. If you create a group for Googlebot, the general bans no longer apply to it: repeat them in its group.
• A path without a leading “/”, or a stray capital letter: the rule does not target what you think.

Frequently asked questions