Site configuration checks
Nine checks that run once per host rather than once per URL, because they describe the site rather than a page: whether http still redirects to https, whether the non-canonical hostname still redirects, whether a missing page still returns a real 404, whether the sitemap and robots.txt are still reachable, and what robots.txt now says. Being host-scoped, they cost almost nothing and do not count against your URL limit.
Soft 404s are the expensive one
A soft 404 is a missing page that returns HTTP 200 with a "not found" message in the body. Google indexes it as a real page, finds it thin and duplicated across every bad URL on the site, and draws conclusions about the site rather than about the URL.
CrawlWarden detects it by requesting a path that cannot exist and reading the status. It is one extra request per host per run, and it catches an entire class of routing regression - the kind a catch-all route introduces on the day someone adds one.
Both redirect checks exist to stop authority splitting
http:// and https://, www. and bare, are separate URLs to a crawler. If both are reachable and both are indexable, links and ranking signals divide between them and neither version gets the benefit of the other. The redirects that prevent this are set once, at the server or the CDN, and then forgotten - which is exactly the shape of thing that breaks during an infrastructure change and stays broken.
On llms.txt, honestly
The check exists and is off by default, which reflects the evidence. Ahrefs analysed 137,000 domains: 28% publish an llms.txt, 97% of those files received no traffic in a month, and no request ever arrived for one that did not exist - nothing goes looking. If you publish one, watching it change costs nothing. Do not read the check’s existence as a recommendation to create one.
All 9 checks in this group
HTTP to HTTPS redirect
on by default · per host · confirmed over 2 runs
http:// stopped redirecting to https://.
http:// stopped redirecting. Either two indexable copies of the site, or a browser security warning on every page of it.
Canonical host redirect
on by default · per host · confirmed over 2 runs
The non-canonical hostname stopped redirecting.
The non-canonical hostname stopped redirecting. www and bare are two sites to a crawler, and authority divides between them.
Real 404 on a missing page
on by default · per host · confirmed over 3 runs
A missing page stopped returning a real 404.
A missing page started returning 200. Google indexes the soft 404s as real pages and draws conclusions about the site, not the URL.
XML sitemap reachable
on by default · per host · confirmed over 2 runs
The XML sitemap stopped being reachable.
The sitemap stopped being fetchable, usually after a path changed or a generator stopped running.
Sitemap sample returns 200
off by default · per host · confirmed over 4 runs
URLs listed in the sitemap stopped returning 200.
Off by default; costs extra requests. URLs the sitemap lists have stopped resolving, so the file is now pointing crawlers at dead ends.
robots.txt content
on by default · per host · confirmed over 2 runs
The contents of robots.txt changed.
Any edit at all, whether or not it changes the verdict for a URL you watch. This is the file where one line has the largest blast radius on the site.
llms.txt
off by default · per host · confirmed over 4 runs
llms.txt appeared, changed or disappeared.
Off by default. Appeared, changed or disappeared. Worth watching if you publish one; the evidence that publishing one does anything is thin.
Favicon reachable
off by default · per host · confirmed over 4 runs
The favicon stopped being reachable.
Off by default. Cosmetic, and among the first things a stranger notices is missing.
Declared key assets
off by default · per host · confirmed over 3 runs
A CSS or JS file you declared stopped returning 200.
Off by default. A CSS or JS file you named stopped returning 200 - the page still loads, unstyled or inert.
Every check on this page is available on every plan, including the free one - limits are on sites and URLs, never on checks. CrawlWarden pricing, per site rather than per URL.