CrawlWarden

X-Robots-Tag checker

Paste one URL. This fetches it once and shows the X-Robots-Tag response header, the robots meta tags in the HTML, and the canonical, exactly as they came back.

One URL at a time. Fetched with our own user agent, CrawlWarden/1.0, never with Googlebot's. Why that matters.

What this reads, and what it does not

  • It fetches the URL once, follows up to five redirects, and reads the response headers and the head of the HTML.
  • It does not crawl. Nothing but the URL you typed is requested, and no link on the page is followed.
  • It does not fetch as Googlebot. CrawlWarden requests a page a second time with a Googlebot user agent only on properties a customer has declared, at their request, which a URL typed here is not.
  • It reports one moment. A header set by a server rule or an edge worker can differ by region, by path and by day.

Reading the result

X-Robots-Tag carries the same directives as the robots meta tag and never appears in the HTML, so a page can look clean in view-source and still be excluded. Where both are present, Google documents that it combines them and the more restrictive directive applies.

A directive can also name one crawler, written as googlebot: noindex. The raw header value is printed above as it came back, because the split into separate lines is a reading aid and the header itself is the fact.

The longer version, including where the header usually comes from and how it differs from the meta tag, is in the X-Robots-Tag explainer.

This is one URL, once

That is what a checker is. The thing that catches a header appearing on a Tuesday is a check that runs on its own and tells you when the answer changes. CrawlWarden watches this header on every fetch, confirms a change twice before it emails anyone, and is free for one site.

See what the plans cover, or read what else is watched on a page.