3 min read
A page dropped out of my XML sitemap - how much does that matter?
By CrawlWarden, which builds the monitoring described at the foot of this article.
Published , sources last re-read .
By itself, very little: an XML sitemap is a discovery aid, not an indexing instruction, and Google will neither crawl nor index a page because the sitemap says to. What matters is what the disappearance implies. Sitemaps are generated, so a page leaving one means an upstream rule changed - the page was unpublished, excluded by a CMS flag, dropped from a query, or lost its status as indexable. The sitemap is the symptom, and it is often the earliest visible one.
What a sitemap is genuinely useful for
Mostly as a diff tool. The interesting comparison is the set of URLs you published against the set Google reports as indexed, and the ghost URLs in Search Console that appear in neither.
- Confirming that what you think you published is what is actually listed.
- Spotting URLs that should not exist - parameterised duplicates, staging paths, orphaned test pages.
- Feeding IndexNow, which Bing, DuckDuckGo, Yandex and others honour and Google does not. Bing frequently indexes a new site months before Google does, and several assistants ground their answers in Bing.
What it is not useful for
- Making Google index a page. "Crawled - currently not indexed" means Google read the page and chose not to keep it. Resubmitting the sitemap does nothing at all.
- Signalling freshness with
lastmod. Google keeps its own crawl records and stops trusting alastmodthat repeatedly claims a change that did not happen. - Passing authority. An HTML hub page that links to the same URLs does that; an XML file does not.
What to check when a page disappears from one
- Is the page still reachable, and does it still return 200?
- Did it gain a noindex tag or an X-Robots-Tag header? Most sitemap generators exclude non-indexable pages automatically, which makes the sitemap a proxy alarm for the real problem.
- Did its URL change? A slug edit removes the old URL and adds a new one in the same regeneration.
- Is it still linked from anywhere on the site? A page that left the sitemap and has no internal links pointing at it is an orphan.
The checks that report this
CrawlWarden watches a list of URLs you choose and reports what changed. It does not diagnose and it does not score. Severity is a property of the check rather than a setting: the number below is how many consecutive runs must see the new value before it becomes an email, and it is never less than two, because one observation is noise.
Sitemap membershipconfirmed over 4 consecutive runs
Reports when a watched page is no longer listed in the site’s XML sitemap.
Sitemap reachableconfirmed over 2 consecutive runs
Reports when the sitemap itself stops being fetchable.
Every check is available on every plan, including the free one. CrawlWarden pricing, per site rather than per URL.
Questions
- Will Google index a page because it is in my sitemap?
- No. A sitemap suggests URLs for discovery. It carries no indexing authority.
- Should I add lastmod dates?
- Only if they are true. An accurate lastmod is mildly useful; one that updates on every deploy is worse than none, because Google compares it against its own crawl history and stops trusting the file.
- Why does this check take four runs to confirm?
- Sitemaps are regenerated on a schedule and are frequently served from a cache, so a single absent entry is far more likely to be a partial regeneration than a real removal. Four consecutive observations is the threshold where it stops being plausible noise.
Related
Sources
What was checked, and where. An entry with no link is something that could not be verified, named here rather than left out.
Google Search Central: Sitemaps overview
The whole premise of the page: "A sitemap helps search engines discover URLs on your site, but it doesn’t guarantee that all the items in your sitemap will be crawled and indexed."
Google Search Central: Build and submit a sitemap
What belongs in a sitemap and what the formats are, behind the "what a sitemap is genuinely useful for" section.
Google Search Console Help: Sitemaps report
What Search Console reports about a submitted sitemap, and why a drop shows up there before it shows up in traffic.
Not verified: search volume for this query
No keyword volume tool is connected to this project. Demand was judged from a live depersonalised US SERP.
Know the moment this happens to you
Give CrawlWarden a domain, keep the pages it suggests, and the first check runs immediately. Free forever for one site.