3 min read
What is the X-Robots-Tag header, and why can I not see it in the page source?
By CrawlWarden, which builds the monitoring described at the foot of this article.
Published , sources last re-read .
X-Robots-Tag is an HTTP response header that carries the same directives as the robots meta tag - noindex, nofollow, noarchive - but travels in the response headers rather than the HTML body. It is invisible in view-source and invisible in the browser, which is exactly why it is missed. It is typically set by a web server rule, a CDN worker, or a framework middleware, so it can be applied to a whole path prefix by someone who never touched the page.
Where the header usually comes from
- An nginx
add_header X-Robots-Tag "noindex";block scoped to a location that matches more than it was meant to. - An Apache
Header set X-Robots-Tagrule in a.htaccessfile inherited by subdirectories. - A CDN or edge worker - Cloudflare Workers, Vercel middleware, Fastly VCL - applying a rule to a route pattern.
- A framework default. Some hosting platforms add the header automatically to preview and branch deployments, which is correct behaviour until a preview domain is promoted.
It is the only way to noindex a non-HTML file
A PDF, an image, a CSV or a JSON endpoint has no <head> to put a meta tag in. The header is the only mechanism available, which is a genuine and legitimate use - it is also why a blanket rule written for /downloads/*.pdf can catch /downloads/index.html when the pattern is loose.
How to read it
The header is visible in any HTTP client and in the browser devtools Network tab, under the response headers for the document request - not in the Elements panel, which shows the DOM.
curl -sI https://example.com/pageprints every response header, including this one.- Paste the URL into the free checker to read the header, the robots meta tags and the canonical in one fetch, without a terminal. It reads one URL once, with the CrawlWarden user agent.
- In Chrome devtools: Network → reload → click the first document row → Headers → Response Headers.
- Google Search Console URL Inspection reports the resulting verdict but does not name the header as the cause, so a header-set noindex and a tag-set noindex look identical in the report.
- Read it twice, once with your browser user agent and once with Googlebot's. A header or a status applied only to crawlers is invisible to the first fetch, and that case is not rare: in a measurement of 105 public domains, 5.1 percent of the sites healthy to a browser answered a Googlebot user agent with something else (the method, the counts and the limits).
Header versus meta tag
| Robots meta tag | X-Robots-Tag header | |
|---|---|---|
| Effect on indexing | Identical | Identical |
| Visible in view-source | Yes | No |
| Works on PDFs and images | No | Yes |
| Usually set by | The page template or CMS | The server, CDN or middleware |
| Usually changed by | Whoever edits content | Whoever edits infrastructure |
If both are present
Google combines them, and the most restrictive directive wins. A page with index in the meta tag and noindex in the header is noindexed. There is no precedence trick to exploit here - remove whichever one is wrong.
The checks that report this
CrawlWarden watches a list of URLs you choose and reports what changed. It does not diagnose and it does not score. Severity is a property of the check rather than a setting: the number below is how many consecutive runs must see the new value before it becomes an email, and it is never less than two, because one observation is noise.
X-Robots-Tagconfirmed over 2 consecutive runs
Reads the response header on every fetch and reports a change to its value.
Robots metaconfirmed over 2 consecutive runs
Watches the HTML tag alongside it, so a directive arriving by either route is seen.
Every check is available on every plan, including the free one. CrawlWarden pricing, per site rather than per URL.
Questions
- Does X-Robots-Tag override the meta tag?
- Neither overrides the other. Google reads both and applies the most restrictive combination.
- Why does my preview deployment have this header?
- Several hosting platforms add it automatically to preview and branch URLs to keep duplicate environments out of the index. That is the right default. It becomes a problem only when a preview domain is promoted to production without the rule being lifted.
- Can I see the header in view-source?
- No. View-source shows the response body. The header is not in the body.
Related
Sources
What was checked, and where. An entry with no link is something that could not be verified, named here rather than left out.
Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
That "any rule that can be used in a robots meta tag can also be specified as an X-Robots-Tag", that it is the way to reach non-HTML files "where the usage of robots meta tags in HTML is not possible", and that the more restrictive rule wins when the two disagree.
Google Search Central: Block search indexing with noindex
That the header and the meta tag are equivalent ways to express noindex, and that either is only read if the crawler is allowed to fetch the page.
Not verified: which source sets the header most often
The ordering of server rule, CDN worker and framework middleware is from support experience, not from a published survey.
Not verified: search volume for this query
No keyword volume tool is connected to this project. Demand was judged from a live depersonalised US SERP.
Know the moment this happens to you
Give CrawlWarden a domain, keep the pages it suggests, and the first check runs immediately. Free forever for one site.