Indexability checks
These twelve checks answer one question - can a search engine still index this page - and they are the only group CrawlWarden will not let you switch off, because a selection without them leaves the product doing nothing. They cover the HTTP status, the redirect chain, both robots directives, all three canonical failure modes, sitemap membership, robots.txt, and what Googlebot specifically receives when it asks - which is not always what your browser gets.
Two checks beyond the basics
Most of this group is table stakes. Two go further.
Googlebot divergence fetches each page a second time with a Googlebot user agent, from a non-US address, and compares the final status and final URL against the browser fetch. A geo rule, a WAF or a bot-management layer that answers crawlers differently is invisible from your desk and invisible in every tool that checks from one location as one client. It is the check behind the deindexing stories where the site looked healthy the whole time.
Per-user-agent robots.txt evaluation reads the live file the way a specific crawler reads it, for each agent you choose: Googlebot, Bingbot, GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Applebot-Extended. A site can be open to Googlebot and closed to every assistant, and nothing in Search Console will mention it. Other monitors now check AI crawler rules as well. Here the result is kept in the page’s recorded history, so a change to it appears as evidence beside what Google later reports.
These two are easily confused and neither substitutes for the other. Divergence tests how the server behaves toward a crawler. Robots evaluation reads what the site declares it allows.
Two rules that keep the Googlebot check trustworthy
- A fetch that could not get a truthful answer is reported as blocked, never as a divergence. A check that cries wolf on a healthy site is worse than no check.
- The Googlebot user agent is never used to get past anything. It tests how a site treats Googlebot, on a property the customer declared, at the customer’s request. That is stated plainly on the crawler page.
All 12 checks in this group
HTTP status code
on by default · per URL · confirmed over 2 runs
The page stopped returning the status it used to.
A 200 becoming a 404 or a 500 is the page leaving the index on the next crawl. A 200 becoming a 301 is the page moving, whether or not anyone meant it to.
Redirect chain
on by default · per URL · confirmed over 3 runs
The page now redirects somewhere else, or through more hops.
A page that starts redirecting, changes destination, or gains a hop. Long chains cost crawl budget and eventually stop being followed; loops are caught here too.
Googlebot status, from a non-US address
on by default · per URL · confirmed over 2 runs
The status Googlebot gets, fetched from a non-US address.
The status Googlebot actually receives. A geo rule or a firewall can answer it with a 403 while every browser you own sees a healthy page.
Googlebot / browser divergence
on by default · per URL · confirmed over 2 runs
The site answers Googlebot differently from a browser.
The site has started answering a crawler differently from a browser. Accidental almost every time, and indistinguishable from cloaking to the party that decides.
robots meta tag
on by default · per URL · confirmed over 2 runs
A robots meta tag changed - noindex is the one that kills a page.
noindex is the one that ends the page. The tag arrives from a staging default, a CMS checkbox or a template edit, and nothing in a normal deploy objects.
X-Robots-Tag header
on by default · per URL · confirmed over 2 runs
An X-Robots-Tag header changed. Same effect as the meta tag, easier to miss.
The same directive in a response header, invisible in view-source. Usually set by a server rule or an edge worker rather than by anyone editing the page.
rel=canonical value
on by default · per URL · confirmed over 3 runs
The canonical tag now points somewhere else.
The page now names a different URL as the real one. Google usually obeys, consolidates the ranking into that URL, and stops showing this page.
Canonical target resolves
on by default · per URL · confirmed over 3 runs
The canonical target stopped returning 200.
The canonical points at a URL that no longer returns 200. Google falls back to its own choice and the outcome stops being predictable.
Canonical is self-referential
on by default · per URL · confirmed over 3 runs
The page stopped pointing its canonical at itself.
A page that used to declare itself canonical no longer does. Usually a trailing-slash, www or scheme change applied in one place and not the other.
Present in the XML sitemap
on by default · per host · confirmed over 4 runs
The page dropped out of the XML sitemap.
Sitemaps are generated, so a page leaving one means an upstream rule changed. Rarely the problem itself; often the earliest visible symptom of it.
robots.txt reachable
on by default · per host · confirmed over 2 runs
robots.txt stopped being reachable.
A missing robots.txt means "crawl everything" and is fine. One returning a 5xx is read by several crawlers as "crawl nothing" until it clears.
robots.txt disallows this URL
on by default · per host · confirmed over 2 runs
robots.txt now blocks this page for a crawler you watch.
Evaluated per user agent, for each crawler you watch. One line can take a path prefix out of one crawler’s reach and leave the others untouched.
Every check on this page is available on every plan, including the free one - limits are on sites and URLs, never on checks. CrawlWarden pricing, per site rather than per URL.