4 min read
A page went noindex - what happened, and how do I catch it next time?
By CrawlWarden, which builds the monitoring described at the foot of this article.
Published , sources last re-read .
A page "goes noindex" when a robots meta tag saying <meta name="robots" content="noindex"> appears in its HTML. Google drops the page from its index on the next crawl, usually within days, and the page stops receiving search traffic entirely. The tag is almost never added on purpose. It arrives from a staging default that shipped, a CMS checkbox, a plugin, or a template change, and nothing in a normal deploy pipeline objects.
The four ways a noindex gets onto a live page
- A staging environment sets noindex site-wide, and a deploy promotes the staging config to production. This is the most common cause by a wide margin.
- A CMS toggle. WordPress has "Discourage search engines from indexing this site" in Settings → Reading; Shopify, Webflow and Squarespace each have their own version.
- An SEO plugin rule. A pattern meant for one archive template matches more paths than intended after a URL change.
- A template edit. A shared layout gains a conditional robots tag for one route and the condition is wrong.
noindex, nofollow, robots.txt and 410 do different jobs
These are routinely confused, and picking the wrong one is how pages either stay indexed when you wanted them gone or disappear when you did not.
| Directive | What it does | Use it when |
|---|---|---|
noindex meta tag | Google crawls the page, reads the tag, and removes the URL from the index. Crawling continues. | A page must stay reachable to users but must not rank. |
nofollow | A hint about the links on the page. Has nothing to do with whether the page itself is indexed. | Rarely. It is not an indexing control. |
Disallow in robots.txt | Blocks crawling, not indexing. A blocked URL can still appear in results from links alone, and Google cannot see a noindex tag it is forbidden to fetch. | You want to save crawl budget, not remove a page. |
| HTTP 410 Gone | Definitive on first contact. No re-fetch cycle needed. | Removing pages in bulk - expired listings, pruned templates, dead SKUs. |
Why it is usually found late
Nothing breaks visibly. The page renders, the links work, the analytics tag fires, and the deploy is green. Google Search Console will eventually report the page as "Excluded by ‘noindex’ tag", but the Page Indexing report is a weekly-ish habit at best and the data itself lags around three days.
The gap between the deploy and the discovery is where the loss is. A pricing page that goes noindex on a Friday and is noticed at the next monthly review has been out of the index for four weeks.
How to verify it right now
- Fetch the page without a browser:
curl -s https://example.com/page | grep -i "robots". This also reveals a tag that JavaScript would have hidden from view-source. - Check the HTTP headers too -
curl -sI https://example.com/page | grep -i x-robots-tag. AnX-Robots-Tag: noindexheader has identical effect and does not appear in the HTML at all. - In Google Search Console, use URL Inspection → Test Live URL, and read the "Indexing allowed?" line rather than the cached verdict.
The checks that report this
CrawlWarden watches a list of URLs you choose and reports what changed. It does not diagnose and it does not score. Severity is a property of the check rather than a setting: the number below is how many consecutive runs must see the new value before it becomes an email, and it is never less than two, because one observation is noise.
Robots metaconfirmed over 2 consecutive runs
Reads the robots meta tag on every fetch and reports any change to its content, noindex included.
X-Robots-Tagconfirmed over 2 consecutive runs
Reads the response header, which carries the same directive and is easier to miss.
Every check is available on every plan, including the free one. CrawlWarden pricing, per site rather than per URL.
Questions
- How long does it take Google to drop a noindexed page?
- It happens on the next crawl of that URL. For a frequently-crawled page that is often within a day or two; for a page Google visits monthly it can be weeks. There is no way to make it faster, and no reason to want to.
- Does removing the tag bring the page back?
- Usually, yes, and usually to roughly where it was - but only after Google re-crawls and re-processes the page. Requesting indexing in Search Console shortens the wait. Rankings can take longer to settle than indexing does.
- Can I just block the page in robots.txt instead?
- No. That blocks crawling, not indexing, and it prevents Google from ever seeing a noindex tag on the page. The two directives are frequently used as if they were interchangeable; they are close to opposites.
- Will CrawlWarden tell me why the tag appeared?
- No. It reports that the value changed, from what to what, and when - twice confirmed. Diagnosis is deliberately out of scope.
Related
- the X-Robots-Tag header version of the same problem
- the state Google gives when nothing is blocking the page and it still is not indexed
- robots.txt blocking Googlebot
- a page dropping out of the XML sitemap
- the staging environment the tag usually arrives from
- the full triage order when a page vanishes
- every failure mode CrawlWarden watches for
- every check it runs, and what a change in it means
Sources
What was checked, and where. An entry with no link is something that could not be verified, named here rather than left out.
Google Search Central: Block search indexing with noindex
The meta tag syntax, that Google drops the page the next time it crawls it, and that the page has to stay crawlable for the tag to be read at all.
Google Search Central: Robots meta tag, data-nosnippet, and X-Robots-Tag specifications
The full directive list, and the rule that "in the case of conflicting robots rules, the more restrictive rule applies".
Google Search Console Help: Page Indexing report
The "Excluded by noindex tag" status named in the verification section.
Not verified: how fast "on the next crawl" is in practice
Google documents that the page is dropped when it is next crawled, not how long that takes. The "usually within days" phrasing is the common observation, not a Google figure.
Not verified: search volume for this query
No keyword volume tool is connected to this project. Demand was judged from a live depersonalised US SERP.
Know the moment this happens to you
Give CrawlWarden a domain, keep the pages it suggests, and the first check runs immediately. Free forever for one site.