Check a site’s llms.txt file

How to use the tool

Enter the address of a site or one of its pages and click “Check”. The tool looks for the llms.txt of the page’s folder, then the one at the root, and keeps the first that answers. It shows the findings, sorted into “To fix”, “To note” and “OK”, the file’s sections with their links, and the file itself, with flagged lines highlighted.

The file’s first ten links are queried at the same time, without following redirects: a link that answers with an error or redirects is flagged. To review a file before putting it live, open “Test an llms.txt before publishing it”: the analysis then runs in your browser.

What an llms.txt is

It is a proposal by Jeremy Howard, published on September 3, 2024: a Markdown file placed at the root of a site, or in one of its subfolders, that presents the site and points to its useful pages, for language models.¹ When several files apply, the most specific one wins.¹

The proposal sets a precise order. An H1 heading with the name of the site or project, the only required section. Then a blockquote that sums up the essentials, details without headings, and H2 sections that hold link lists, each written “[name](url)” and followed if needed by a colon and a note. An H2 section named “Optional” gathers the links an agent can skip when it needs less context.¹

# Site name

> What the site does, in one or two sentences.

Useful details, without a heading.

## Main pages

- [Services](https://www.example.com/services/): what is offered
- [Contact](https://www.example.com/contact/)

## Optional

- [Legal notice](https://www.example.com/legal/)

The same proposal suggests publishing a Markdown version of useful pages, at the same address followed by “.md”.¹ The tool does not check these versions.

What the tool checks

First, the server’s answer. A 404 means there is no file, which is not a fault. A 5xx error is the only case where Lighthouse’s llms.txt audit fails.⁴ A trap worth watching is the HTML page served in place of the file, with a 200 status: the site returns its home page or its error page for any unknown address, and a bot then receives HTML instead of text.

Then the structure: an H1 that opens the file, a blockquote summary, H2 sections, list items that each start with a link, no heading of level 3 or more, no duplicate link. Finally the links themselves: readable addresses, and the HTTP status of the first ten.

What Google and Lighthouse do with it

Google says it plainly in its guide on generative AI: you do not need to create AI text files such as llms.txt to appear in Search, and Google Search does not use them. Keeping one neither harms nor helps visibility or rankings; Google adds that you may maintain one for other services that use these files.² Its page on AI features says the same.³

Lighthouse, Chrome’s auditing tool, still checks the file in its Agentic Browsing category, which assesses how AI agents can navigate a site.⁵ The audit is marked not applicable when the file is missing, since the file remains optional, and only fails on a server error.⁴

Should you publish one?

For Google, no: its documentation says so in writing.² For other agents, the proposal targets models that consult a site while answering, and a short, up-to-date, well-built file costs little. If one exists, it might as well be readable: a broken file, an HTML page in its place or dead links give a poor picture of the site to whoever reads it.

An llms.txt controls no access. It neither blocks nor allows any bot: that is the role of robots.txt, which crawlers read before downloading a page.

What the tool does not see

The tool checks the form of the file, not the relevance of its content: it does not say whether the chosen pages are the right ones or whether the summary is accurate. It does not read the Markdown versions of pages, only goes up one folder level before the root, reads the file up to 500 KiB and checks ten links at most.

Each check counts as one request against a quota of 20 per 10-minute window and per visitor, shared with the site’s other tools. The server reads the file, returns the result and keeps neither the address nor the content.

After the test

An llms.txt only matters if bots can read the site. The AI crawler access test shows what robots.txt allows GPTBot, ClaudeBot or PerplexityBot, and the bot IP checker tells you whether the ones in your logs are genuine. To get cited by generative engines, content and crawlability weigh more than a file: that is the subject of the article on GEO.

Frequently asked questions