4 min read

A page went noindex - what happened, and how do I catch it next time?

By CrawlWarden, which builds the monitoring described at the foot of this article.

Published , sources last re-read .

A page "goes noindex" when a robots meta tag saying <meta name="robots" content="noindex"> appears in its HTML. Google drops the page from its index on the next crawl, usually within days, and the page stops receiving search traffic entirely. The tag is almost never added on purpose. It arrives from a staging default that shipped, a CMS checkbox, a plugin, or a template change, and nothing in a normal deploy pipeline objects.

The four ways a noindex gets onto a live page

  • A staging environment sets noindex site-wide, and a deploy promotes the staging config to production. This is the most common cause by a wide margin.
  • A CMS toggle. WordPress has "Discourage search engines from indexing this site" in Settings → Reading; Shopify, Webflow and Squarespace each have their own version.
  • An SEO plugin rule. A pattern meant for one archive template matches more paths than intended after a URL change.
  • A template edit. A shared layout gains a conditional robots tag for one route and the condition is wrong.

noindex, nofollow, robots.txt and 410 do different jobs

These are routinely confused, and picking the wrong one is how pages either stay indexed when you wanted them gone or disappear when you did not.

DirectiveWhat it doesUse it when
noindex meta tagGoogle crawls the page, reads the tag, and removes the URL from the index. Crawling continues.A page must stay reachable to users but must not rank.
nofollowA hint about the links on the page. Has nothing to do with whether the page itself is indexed.Rarely. It is not an indexing control.
Disallow in robots.txtBlocks crawling, not indexing. A blocked URL can still appear in results from links alone, and Google cannot see a noindex tag it is forbidden to fetch.You want to save crawl budget, not remove a page.
HTTP 410 GoneDefinitive on first contact. No re-fetch cycle needed.Removing pages in bulk - expired listings, pruned templates, dead SKUs.

Why it is usually found late

Nothing breaks visibly. The page renders, the links work, the analytics tag fires, and the deploy is green. Google Search Console will eventually report the page as "Excluded by ‘noindex’ tag", but the Page Indexing report is a weekly-ish habit at best and the data itself lags around three days.

The gap between the deploy and the discovery is where the loss is. A pricing page that goes noindex on a Friday and is noticed at the next monthly review has been out of the index for four weeks.

How to verify it right now

  • Fetch the page without a browser: curl -s https://example.com/page | grep -i "robots". This also reveals a tag that JavaScript would have hidden from view-source.
  • Check the HTTP headers too - curl -sI https://example.com/page | grep -i x-robots-tag. An X-Robots-Tag: noindex header has identical effect and does not appear in the HTML at all.
  • In Google Search Console, use URL Inspection → Test Live URL, and read the "Indexing allowed?" line rather than the cached verdict.

The checks that report this

CrawlWarden watches a list of URLs you choose and reports what changed. It does not diagnose and it does not score. Severity is a property of the check rather than a setting: the number below is how many consecutive runs must see the new value before it becomes an email, and it is never less than two, because one observation is noise.

  • Robots metaconfirmed over 2 consecutive runs

    Reads the robots meta tag on every fetch and reports any change to its content, noindex included.

  • X-Robots-Tagconfirmed over 2 consecutive runs

    Reads the response header, which carries the same directive and is easier to miss.

Every check is available on every plan, including the free one. CrawlWarden pricing, per site rather than per URL.

Questions

How long does it take Google to drop a noindexed page?
It happens on the next crawl of that URL. For a frequently-crawled page that is often within a day or two; for a page Google visits monthly it can be weeks. There is no way to make it faster, and no reason to want to.
Does removing the tag bring the page back?
Usually, yes, and usually to roughly where it was - but only after Google re-crawls and re-processes the page. Requesting indexing in Search Console shortens the wait. Rankings can take longer to settle than indexing does.
Can I just block the page in robots.txt instead?
No. That blocks crawling, not indexing, and it prevents Google from ever seeing a noindex tag on the page. The two directives are frequently used as if they were interchangeable; they are close to opposites.
Will CrawlWarden tell me why the tag appeared?
No. It reports that the value changed, from what to what, and when - twice confirmed. Diagnosis is deliberately out of scope.

Related

Sources

What was checked, and where. An entry with no link is something that could not be verified, named here rather than left out.

Know the moment this happens to you

Give CrawlWarden a domain, keep the pages it suggests, and the first check runs immediately. Free forever for one site.