Read and analyse the HTTP headers of a page

How to use the tool

Paste the URL of a page, a PDF or an image and click “Analyse”. The tool sends a GET request, follows up to ten redirects and shows the chain it got: the status code and URL of each step, then the final response. The table then lists the headers that matter, sorted into four groups: SEO and crawling, cache, compression, security.

Each row has a status. “Present” means the header is there and raises no known issue, “Missing” that it is absent, which is sometimes fine, and “Review” that it is there but its value deserves a look. The right-hand column says why, with the number of the source. The tool computes no overall score: a missing security header matters little on a brochure page without forms, a lot on a customer account area. Below the table you get the full list of headers received, in order and duplicates included.

Headers that matter for SEO

Headers are invisible in the page, but Googlebot reads them before anything else. The one with the heaviest consequences is X-Robots-Tag. Google accepts in this header every rule of the robots meta tag: noindex, nofollow, nosnippet, max-image-preview and the others. Several X-Robots-Tag headers can be combined, one line can target a specific crawler (“googlebot: noindex”), and when rules conflict, the more restrictive one applies.¹ A noindex set here by a forgotten server setting removes the page from results with nothing showing in the HTML.

These rules are only read if robots.txt lets the crawler fetch the page.¹ Blocking a page in your robots.txt rules while giving it a noindex therefore makes that noindex invisible.

The Link header can carry a canonical: “Link: <https://www.example.com/white-paper.pdf>; rel="canonical"”. Google supports it for web search, including for files without HTML such as PDFs, and asks for an absolute URL, as in the tag. It also recommends picking a single method: giving one canonical in the header and another in the HTML is error-prone.²

Content-Type declares the document type and, through its charset parameter, its character encoding, for example “text/html; charset=utf-8”.³ Vary lists the request headers that change the response; caches keep one version per value, and “Vary: *” makes the response uncacheable.⁴ Google ties “Vary: user-agent” to dynamic serving, where one URL serves different HTML to mobile and desktop.⁵

ETag, Last-Modified and conditional crawling

In a December 2024 post, Gary Illyes of Google reports that about 0.026% of Google’s fetches were cacheable ten years ago, against 0.017% today. He also explains that allowing caching may help a large site with rarely changing content get crawled more efficiently.⁶

The mechanism relies on two pairs of headers. The server sends an ETag, a fingerprint of the page version; on the next visit, the crawler sends it back in If-None-Match. If nothing changed, the server answers 304 Not Modified with no body: it neither builds the page nor sends it. Last-Modified and If-Modified-Since do the same with a date. Google recommends ETag, which is less error-prone since its value has no imposed structure, and suggests setting both when possible. For Last-Modified, it recommends the “Fri, 4 Sep 1998 19:15:56 GMT” format and suggests adding Cache-Control: max-age, the number of seconds the content should stay unchanged.⁶ When both headers are present, Google’s crawlers use the ETag.⁷

Google leaves it to the site to decide what counts as a change: according to the same post, updating the copyright year at the bottom of the page is probably not significant enough for a new ETag.⁶

Caching: Cache-Control, Expires, Age and CDNs

Cache-Control carries the caching directives. max-age sets the freshness lifetime in seconds, s-maxage overrides it for shared caches such as a CDN, no-cache allows storage but requires revalidation before each use, no-store forbids any storage, private keeps the response to the browser.⁸ web.dev keeps the one-year lifetime (max-age=31536000) for files whose URL changes with each version, such as a stylesheet with a fingerprinted name, and suggests no-cache for resources that must be revalidated before every use.⁹ An HTML page keeps its URL: caching it for a year risks returning visitors not seeing its updates. The tool flags it.

Expires gives an expiry date, and it is ignored as soon as Cache-Control contains max-age or s-maxage.¹⁰ Age gives how many seconds the copy has been held by an intermediate cache; a value of 0 usually means the response was just fetched from the origin server.¹¹

CDNs and hosts add their own headers: cf-cache-status at Cloudflare, x-cache or x-cache-status elsewhere. At Cloudflare, HIT means the resource was in the cache, MISS that it was fetched from the origin, DYNAMIC that it was deemed not eligible for caching.¹² The same site can answer MISS then HIT a few seconds apart: run the analysis again to watch the cache fill.

Compression

Content-Encoding says how the response body was compressed. MDN recommends compressing as much as possible, except for already compressed formats such as JPEG or ZIP, which compressing again can make larger.¹³ Google’s crawlers accept gzip, deflate and Brotli (br), and advertise what they can read in their Accept-Encoding header.⁷ The tool also asks for a compressed response: an HTML page that comes back without Content-Encoding is served uncompressed, which makes it heavier for visitors and crawlers alike.

Security headers

These headers tell the browser how to protect the people visiting the page. The OWASP Secure Headers Project maintains a reference configuration and the list of headers that became obsolete; the tool follows its recommendations.¹⁴

• Strict-Transport-Security (HSTS). Forces the browser to come back over HTTPS for max-age seconds. Sent over HTTP, it is ignored. To be accepted into browser preload lists, max-age must be at least one year, with includeSubDomains.¹⁵ OWASP suggests two years.¹⁴
• Content-Security-Policy (CSP). Limits the sources the page may load, which guards against cross-site scripting. The Report-Only variant reports violations without blocking anything: it is meant for testing.¹⁶
• X-Content-Type-Options. Its only value, nosniff, makes the browser follow the declared Content-Type and refuse a script or stylesheet served with the wrong type.¹⁷
• X-Frame-Options or frame-ancestors. Stop another site from embedding the page in a frame to hijack clicks. X-Frame-Options accepts DENY and SAMEORIGIN; with ALLOW-FROM, modern browsers ignore the whole header.¹⁸ The CSP frame-ancestors directive does the same job with more options.¹⁹
• Referrer-Policy. Controls how much of its address the page passes on to the sites it links to. Without this header, browsers apply strict-origin-when-cross-origin; unsafe-url sends the full address of HTTPS pages, even to HTTP sites.²⁰
• Permissions-Policy. Allows or denies browser features such as the camera or geolocation to the page and its frames.²¹ Its predecessor, Feature-Policy, is obsolete.¹⁴

The tool also lists the cookies the page sets. Their value is never shown, only their attributes: Secure (sent over HTTPS only), HttpOnly (unreadable by page JavaScript) and SameSite, whose None value requires Secure.²² Finally, it flags X-XSS-Protection, which OWASP lists as obsolete and advises setting to 0, and the headers that reveal the server technology or version, such as X-Powered-By, which OWASP advises removing.¹⁴

What the tool sees, and what it does not

The request leaves from the nemociment.fr server, under a user-agent that presents it as a tool and not as Googlebot. A site that answers differently depending on the crawler, the country or a firewall may therefore send other headers to Google. Only the headers of the final response are analysed; intermediate redirects show up only through their status code. Rules set in the HTML, robots meta tag or canonical tag, are not read here.

Reading headers is part of the technical check of a website before working on its content: a noindex sent by the server is enough to take a page out of the results, whatever its content. On a whole site, with its page templates, files and server rules, this review is done template by template, and it is one of the first steps of the audit I run for my clients.

Frequently asked questions