Indexability checker: can Google index your page?

How to use the tool

Paste the full URL of a page and click “Check”. The tool requests the page the way a browser would, follows redirects to the last one, reads the robots.txt of the host it lands on and goes through everything that can hold Google back. The verdict comes in three words: indexable, not indexable with the blocking cause, or check when nothing forbids indexing but a signal may get in the way.

Under the verdict, every check shows its evidence: the status code, the meta tag as written, the header received or the robots.txt line that applies. That line is also highlighted in the file, shown at the bottom of the result. The tool reads the HTML as the server sends it, without running JavaScript.

Crawling and indexing are two separate steps

Google first crawls a URL, meaning it downloads it, then decides whether to add it to its index. The signals act at different moments. robots.txt works on the first step: it tells the crawler what it may download. The noindex rule works on the second: it asks for the page to be left out of search results.¹

Mixing the two up is the most common mistake. Google itself writes that robots.txt is not a way to keep a page out of its results: if other pages link to it, its URL can still appear, without a description.³ To get a page out of the index you need noindex, and that noindex has to be readable, so the page must not be blocked by robots.txt.¹

What the tool checks

• Final HTTP status. A 2xx page has its content considered for indexing. A 4xx error, except 429, drops the URL from the index; a 5xx error slows crawling and, if it lasts, removes the URL as well.⁵
• Redirect chain. Google’s crawlers follow up to 10 redirect hops.⁵ Each step is listed with its code and destination. A URL that redirects will not be indexed itself: the analysis covers the page it lands on.
• robots.txt for Googlebot. The tool applies the rules of Google’s specification: the group named after the crawler or, failing that, the “*” group, with the longest rule winning. A missing file (4xx) blocks nothing; a 5xx error makes Google stop crawling the site for 12 hours, after which it uses the last cached version.⁴
• Robots and googlebot meta tags. “noindex” and “none” exclude the page; “unavailable_after” sets a date after which it no longer shows. When two rules conflict, the more restrictive one applies.²
• X-Robots-Tag header. It has the same effect as the meta tag¹ and can target one crawler, for example “X-Robots-Tag: googlebot: noindex”.² Only values aimed at Googlebot or at all crawlers count here.
• Canonical. The <link rel="canonical"> tag and the HTTP Link header. The tag is only accepted in the <head>, and Google asks you not to declare several different canonicals for the same page.⁶
• Content type. The Content-Type header says whether the response is HTML. For a PDF or an image, only the status code, robots.txt and headers are checked.
• Served text. The tool counts the words in the HTML received, with scripts and styles removed. Below 50 words, a threshold set by the tool, the page is flagged: either it is empty or its content is built by JavaScript.
• hreflang. Declared language versions are listed for information: they steer which version shows in results, without deciding indexing.⁸

The blockers that come up most

A noindex hidden behind robots.txt. A page blocked by robots.txt is not downloaded, so Google sees neither its meta tags nor its headers, which are ignored.² The noindex added to take it out of the index has no effect, and the URL can keep showing. The tool flags it when both are present together.

A noindex removed with JavaScript. Google processes pages in three phases: crawling, rendering, indexing. When it finds a noindex in the original HTML, it may skip rendering, so a script that removes the tag guarantees nothing. If the page should be indexed, the noindex must not be in the original code.⁷

A canonical pointing elsewhere. Strictly speaking it does not prevent indexing, but it is a strong signal that the other URL should become the canonical version.⁶ It is often left over from a copied page template, or from a plugin that writes the home page URL everywhere. The tool rates this case “check”.

A page that is empty without JavaScript. Google queues pages that return 200 for rendering, then runs their JavaScript; content missing from the rendered HTML cannot be indexed. Google still recommends server-side rendering or pre-rendering.⁷

What the tool cannot know

An indexable page is not an indexed page. Search Console’s Page indexing report says so: do not expect every URL on your site to be indexed.⁹ Several of the states it shows are beyond any technical check:

• Discovered, currently not indexed. Google knows the URL but has not crawled it yet.
• Crawled, currently not indexed. Google downloaded it without adding it to the index, and may do so later.
• Duplicate, Google chose different canonical than user. The declared canonical is valid, but Google prefers another one.
• Soft 404. The page returns 200 but its content looks like an error, such as an empty page.⁵

Only Search Console’s URL Inspection tool reports on the indexed version of a page. Its live test checks that Google-InspectionTool can access the page, and Google states that no test guarantees inclusion in the index; the test looks at neither sitemaps nor referring pages.¹⁰ This tool is there to find a technical blocker fast, on any page, including pages of a site you have no access to.

After the check

An indexable page still has to be found. Internal links and the sitemap take care of that; what Google says about XML sitemaps today helps you decide what to put in it and what is pointless. If robots.txt is the blocker, the AI crawler access tool applies the same file to GPTBot, ClaudeBot, PerplexityBot and the other crawlers of generative engines. And when dozens of pages stay out of the index with no visible cause, it is time for a full SEO audit, which cross-checks Search Console reports, a crawl of the site and its server logs.

Frequently asked questions