Analyse your server logs: Googlebot and AI crawlers

How to use the tool

Get your site’s access log from your host’s dashboard or from the server, then drop it in the area above, as it is or compressed as .gz, and click “Analyse”. For a quick test, open “Paste lines instead”. The file is read as a stream: a log of several hundred megabytes goes through, just more slowly.

The result lists each recognised bot: number of requests, share verified by IP address, share of 4xx and 5xx errors, last visit. Then come Googlebot’s details (smartphone crawler, visits per day, most crawled pages, errors served), each bot’s reads of robots.txt, the sitemap and llms.txt, and the list of fake bots.

What an access log contains

Each line describes one request received by the server. In Apache’s “common” format, it gives the client address, the date, the request line with the method and the address requested, the status code and the size sent. The “combined” format adds the referring page and the user agent, the name the client gives itself.¹ nginx’s “combined” format has the same shape.²

It is the only complete record of what bots do on a site: every visit is there, on every address, with the status returned. Search Console gives an overview from Google’s side; the log also shows Bing, AI crawlers and those posing as them.

Recognising bots, and the real ones

A bot is first recognised by its user agent: Google publishes Googlebot’s for smartphone and desktop, along with the names of its other crawlers, from Googlebot-Image to Google-InspectionTool.³ OpenAI, Anthropic and Perplexity publish the names of theirs as well.⁶⁷⁸ Google states that Google-Extended has no user agent of its own: it never shows up in a log, its rules go through the existing crawlers.³

A user agent can be copied freely. So the tool checks the address of every request against the address lists the operators publish, the same ones as the bot IP checker: Google, Microsoft, OpenAI, Anthropic, Perplexity, Apple, Common Crawl, DuckDuckGo and Amazon. Google presents matching against its published ranges as one way to verify its crawlers.⁴ A request that claims to be Googlebot from another address is counted as fake and kept out of the real Googlebot’s statistics. Bots whose operator publishes no list, such as Meta, Mistral AI or SEO tools, are flagged “not verifiable”.

What to look at

Errors served to Googlebot. Google states that a 404 or 410 on a removed page is a strong signal to stop recrawling it, that a soft 404 page keeps being crawled and wastes crawl budget, and that 5xx server errors lower the crawl capacity granted to the site.⁵ A list of 404 addresses that Googlebot visits often calls for redirects or for cleaning up the links that lead there.

The most crawled pages. They show where Googlebot spends its time. Parameter addresses, filter pages or old content at the top of the ranking can point to internal linking or a sitemap to rework, just like important pages missing from it.

AI crawlers. The table shows whether GPTBot, OAI-SearchBot, ClaudeBot or PerplexityBot really visit, and the control files table whether they read robots.txt, the sitemap or llms.txt. It is the most direct way to check what your robots.txt allows them in practice.

What the tool does not see

The tool reads Apache and nginx formats, “common” and “combined”. A custom format, an IIS log or a JSON export is not recognised: unreadable lines are counted and an example line is shown. Without a user agent, in the “common” format, no bot can be recognised.

Behind a CDN or a proxy, the address logged is the proxy’s, not the bot’s, as Apache’s documentation points out:¹ address verification then no longer works, and every verifiable bot will look fake. Rankings track 20,000 distinct addresses per bot at most. The log is sent nowhere; only fetching the address lists counts as a request, against a quota of 20 per 10-minute window.

After the analysis

A suspicious address can also be checked one by one, with its reverse DNS, in the bot IP checker. Errors served to Googlebot are often fixed with redirects, to check afterwards with the redirect checker. Reading logs regularly is part of the technical SEO audit I run for my clients, as the checklist for auditing your site explains.

Frequently asked questions