8 min read
A page disappeared from Google - how do I find out why?
By CrawlWarden, which builds the monitoring described at the foot of this article.
Published .
First confirm the page is really gone rather than ranked lower. Search site: the exact URL with SafeSearch off, and run URL Inspection in Search Console. If it is missing from both, the cause is almost always one of four technical things: a noindex directive, a robots.txt rule, a status code that is no longer 200, or a canonical naming a different URL. Manual actions and security issues are the two deliberate causes, and each has its own report.
Step one: confirm the page is actually missing
A page that fell from position 3 to position 29 produces the same panic and needs a completely different response, so this step is not a formality. Google's own checklist starts in the same place (Why is my page missing from Google Search?).
- Turn SafeSearch off. Google lists this first, because a filtered result set looks identical to a missing page.
- Search
site:url_of_pagefor the single URL, andsite:your_domain_namefor the whole site. If the site returns results and the page does not, you have a page problem rather than a site problem, which halves the list below. - Run URL Inspection on the URL in Search Console, then Test Live URL. The live test is what tells you the current state; the cached verdict tells you the state at the last crawl, and in an incident those are different answers.
- Open the Performance report and filter to that URL. Impressions that fell to zero on a date are a disappearance. Impressions that held while clicks fell are a ranking change, and nothing on this page applies.
Dropped out, or ranked down
Google publishes a taxonomy of traffic drops by the shape of the chart, which is the fastest way to classify what you are looking at before touching anything (Debugging drops in Google Search traffic).
| What the chart does | What it usually is | Where to look first |
|---|---|---|
| Falls off a cliff, site-wide, on one date | A technical change, a manual action, or a security issue. Google: "Technical issues are errors that can prevent Google from crawling, indexing, or serving your pages to users." | Page indexing report, Manual actions, Security issues. In that order. |
| Falls off a cliff on a few URLs only | A page-level directive or status change, usually shipped with a release. | URL Inspection on one affected URL, live test. |
| Steps down and stays down | An algorithmic or competitive change. The pages still exist and still rank, lower. | Performance report, average position by query. |
| Drifts down over months | Interest changing, or the pages ageing. Google lists seasonality separately for a reason. | Compare year on year before concluding anything. |
| Recovers on its own within a day or two | A reporting glitch. Google names this case in its own documentation. | Nothing. Wait. |
The four technical causes, in the order worth checking
Each of these takes one command to rule out, and the order is by how often it is the answer. Run all four before forming a theory.
| Cause | How to check in one line | What Search Console calls it |
|---|---|---|
A noindex directive | curl -s https://example.com/page | grep -i robots and curl -sI https://example.com/page | grep -i x-robots-tag | "Excluded by ‘noindex’ tag". The meta tag version and the header version behave identically and the header is invisible in view-source. |
| A robots.txt rule | curl -s https://example.com/robots.txt and read it the way a crawler does, longest matching rule first | "Blocked by robots.txt". Worth knowing that this blocks crawling rather than indexing, so it produces a listed page with no description as often as a missing one. |
| The status code stopped being 200 | curl -s -o /dev/null -w "%{http_code}" https://example.com/page | "Not found (404)", "Server error (5xx)", "Blocked due to access forbidden (403)", "Page with redirect". A 200 that renders a not-found view is a soft 404 and has its own status. |
| The canonical names another URL | curl -s https://example.com/page | grep -i canonical | "Alternate page with proper canonical tag", or "Duplicate, Google chose different canonical than user". Canonical drift consolidates the page into another URL, which looks exactly like disappearing. |
If all four are clean
Then it is one of the causes Google lists that are not about your markup. The first two are deliberate decisions by Google and each has its own report in Search Console; the rest are ordinary and uninteresting, which is why they get overlooked.
- A manual action. Check the Manual actions report. Google: "If your site doesn't comply with the Spam policies for Google web search, your content might rank lower in results or not appear in results at all."
- A security issue. Check the Security issues report. Google warns users before they reach a flagged site, which collapses traffic without removing the page.
- The page is new. Google notes it can take weeks before a new page is crawled, and a page that was never indexed has not disappeared.
- Duplicate consolidation. Where several URLs serve the same content, only the canonical version appears, whether or not you chose it.
- Findability. A page nothing links to is one Google may simply not reach, which is also why a page falling out of the XML sitemap matters more on a large site than on a small one.
- A crawler-only block. If the page loads for you and 403s for Googlebot, nothing in your HTML will show it. That is an edge rule, and URL Inspection's live test is the quickest way to see it.
What to do once you know
- Fix the cause first. Requesting indexing on a page that still carries a noindex achieves nothing except a queue position.
- Then request indexing through URL Inspection for the handful of URLs that matter. For a site-wide fix, do not request thousands; Google will re-crawl on its own and the tool is rate-limited by design.
- Expect recovery to take as long as a re-crawl takes, which is days for a frequently-crawled page and weeks for one Google visits monthly. Rankings settle more slowly than indexing does.
- Write down the date the cause shipped. The gap between that date and the date you noticed is the only number in this incident you can actually improve.
Why it took three weeks to notice
Every cause in the first table is a one-line change that a browser cannot see. The page renders, the links work, the analytics tag fires, and the deploy is green. Search Console reports all of it eventually, with the Page indexing report lagging a few days and being a habit nobody keeps daily.
So the usual shape of this incident is not a hard problem badly diagnosed. It is a trivial problem found late, and the cost is the number of days in between multiplied by whatever the page was worth. That gap is the thing worth engineering away, and it is the one part of the problem a report cannot fix.
The checks that report this
CrawlWarden watches a list of URLs you choose and reports what changed. It does not diagnose and it does not score. Severity is a property of the check rather than a setting: the number below is how many consecutive runs must see the new value before it becomes an email, and it is never less than two, because one observation is noise.
robots meta tagconfirmed over 2 consecutive runs
Reads the robots meta tag on every fetch and reports any change to its content, noindex included.
X-Robots-Tagconfirmed over 2 consecutive runs
The header carrying the same directive, which never appears in the HTML and is the one people miss.
HTTP status codeconfirmed over 2 consecutive runs
Reports when a URL you watch stops returning the status it used to, in either direction.
rel=canonical valueconfirmed over 3 consecutive runs
Reports when the page starts naming a different URL as the real one, which is how a page disappears while still returning a healthy 200.
Googlebot status, from a non-US addressconfirmed over 2 consecutive runs
The status Googlebot actually receives, fetched separately. A block that applies only to crawlers is invisible to every other check on this list.
Every check is available on every plan, including the free one. CrawlWarden pricing, per site rather than per URL.
Questions
- The page is gone from `site:` but Search Console says it is indexed. Which is right?
- Neither is wrong; they answer different questions.
site:is a live query against what is being served, and it is unreliable at low volumes. Search Console reports the last crawl. Use URL Inspection with Test Live URL, which is the only one of the three that reflects the page as it is right now. - How long until a fixed page comes back?
- It returns on the next crawl of that URL, which is days for a page Googlebot visits often and weeks for one it visits monthly. Requesting indexing shortens the wait for a few URLs. Rankings usually settle a little after indexing does, and not always at exactly the old position.
- Could Google have simply decided not to index it?
- Yes, and Search Console says so plainly when that is the case: "Crawled - currently not indexed" and "Discovered - currently not indexed" are both status values in the Page indexing report. Neither is an error, and neither has a technical fix. That is a content and internal-linking problem.
- Will CrawlWarden tell me why a page disappeared?
- No. It tells you which of the values above changed, from what to what, and on which day, confirmed over consecutive runs. That is the first half of this page done for you and none of the second half; diagnosis is deliberately out of scope, and the Manual actions and Security issues reports are inside your Search Console account, which no third party can read.
Related
- the noindex tag, which is the most common single cause
- the coverage state Google gives when nothing is blocking the page
- a fall in clicks with no visible change on the site
- a page that answers 200 while reading as missing
- a block that applies to Googlebot and not to you
- every failure mode CrawlWarden watches for
- every check it runs, and what a change in it means
Sources
What was checked, and where. An entry with no link is something that could not be verified, named here rather than left out.
Google Search Console Help: Why is my page missing from Google Search?
The verification steps (SafeSearch, site: queries, URL Inspection) and the list of causes, including manual actions, security issues, blocking, duplicates, findability and new pages.
Google Search Central: Debugging drops in Google Search traffic
The classification of drops by chart shape, the wording on technical issues and spam policies, and the reporting-glitch case.
Google Search Console Help: Page Indexing report
The exact status names quoted in the tables, including "Crawled - currently not indexed" and "Blocked due to access forbidden (403)".
Google Search Central: Block search indexing with noindex
That a page blocked in robots.txt never has its noindex read, and that a revisit can take months.
Not verified: search volume for this query
No keyword volume tool was available. Demand was judged from the live US SERP, which carries no ads and ranks three Reddit threads, two Google Help forum threads and a Stack Exchange question on page one.
Not verified: any specific figure for how long recovery takes
Google states that revisit time depends on how important it considers the page and gives no number. This page gives ranges and says they are ranges.
Know the moment this happens to you
Give CrawlWarden a domain, keep the pages it suggests, and the first check runs immediately. Free forever for one site.