How do I take a site down for maintenance without losing search visibility?

Most advice on this is folklore. Google has a specific page on it, and it is short enough to read in full (Temporarily pause or disable a website). The headline guidance: "If you need to urgently disable the site for 1-2 days, then return an informational error page with a 503 HTTP response status code instead of all content", and "If you need to disable the site for a longer time, then provide an indexable home page as a placeholder for users to find in Search by using the 200 HTTP status code."

The reason 503 works at all is in the status code reference: 5xx and 429 "prompt Google's crawlers to temporarily slow down with crawling", and already-indexed URLs are "preserved in the index, but eventually dropped" (HTTP status codes and network errors). Preserved, then eventually dropped. The whole question is how much of "eventually" you are spending.

If your maintenance mode is implemented at the edge or in middleware, it will almost certainly return 503 for every path, robots.txt included. That one detail turns a two-hour window into something with a thirty-day tail, and Google's instruction on it is a single sentence: "Don't return a 503 HTTP response status code for the robots.txt file because this blocks all crawling."

The mechanics are in the robots.txt specification, and they are worth knowing exactly (How Google interprets the robots.txt specification). If Google finds a robots.txt file but cannot fetch it: "For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file." After that, "for the next 30 days Google will use the last good version", and a 503 in particular "results in fairly frequent retrying". If the error is still there after 30 days and the site is generally available, Google behaves as if there is no robots.txt; if the site has general availability problems, "Google will stop crawling the site".

So a 503 on robots.txt stops crawling of the entire site for twelve hours, whatever else you did right. Note the asymmetry with 4xx: Google "treats all 4xx errors, except 429, as if a valid robots.txt file didn't exist", meaning a missing robots.txt is harmless and an erroring one is not. The file where one line has the largest blast radius is also the file most likely to be caught by a blanket rule.

Google asks for it: "Use the retry-after HTTP header with a best effort date or duration." The header is defined in RFC 9110 and takes one of two forms, an HTTP-date or a number of seconds, and it is the standard companion to a 503 (RFC 9110, section 10.2.3).

Treat it as a courtesy rather than a contract. Google describes the value as best effort and publishes no commitment to re-crawl at that moment, and other clients vary. Sending it is correct and costs nothing; planning the window around the assumption that crawlers will come back at exactly that time is not.

• Serve static HTML for the error page. Google asks for this explicitly, along with inline CSS and base-64 images, to keep the page cheap on both sides.
• Put something useful on it: when you expect to be back, and how to reach support. The page will be seen by customers, not only by crawlers.
• Confirm the status from outside your network rather than trusting the configuration: curl -I -X GET "https://www.example.com/" should answer HTTP/1.1 503 Service Unavailable. This is the check Google's own documentation demonstrates.

A 503 freezes more than crawling. Google warns: "it's not possible for Google's systems to refresh titles, descriptions, metadata, or structured data included on a website if a page returns a 503 HTTP response status code." If you are shipping a title or schema change behind the same maintenance window, it will not be picked up until the pages answer 200 again.

It also does not hide you. The pages stay in the index, which is the point, so a search for your brand still shows them and still sends people to an error page. That is the intended trade, but it is worth knowing before a sale or a launch.

• The maintenance page returns 200. Now every URL on the site serves the same thin page with a healthy status, which is a soft 404 at the scale of the whole site and the worst of the available outcomes.
• robots.txt returns 503 with everything else. Twelve hours of no crawling at all, from a window that was supposed to be twenty minutes.
• Everything redirects to /maintenance. A 302 to a single URL means Google crawls that URL instead, and the one page that gets indexed during the window is the apology.
• Google is explicitly warned off the alternatives: "Don't block the website by returning 403, 404, 410 HTTP status codes, or with a noindex robots meta tag or x-robots-tag HTTP header. This will remove the website's URLs from Google Search." The Search Console removal tool is also named as the wrong tool for a closure.
• The 503 outlives the window. A CDN caches it, a flag is not cleared, or one region behind one edge node keeps serving it. Nobody checks, because the deploy succeeded and the site works from the office.

Everything above assumes you chose the downtime. The expensive version is the 503 you did not schedule: a failing origin behind a healthy edge, a database that drops out under load at 03:00, or a rule that answers only crawlers while every human request succeeds. The last of those is a block that your browser will never show you.

Uptime monitoring catches the first two and misses the third, because an uptime monitor fetches as itself from a known address and is usually allowlisted. What distinguishes a crawler-visibility problem from an outage is that nothing on your dashboard is red.

CrawlWarden watches a list of URLs you choose and reports what changed. It does not diagnose and it does not score. Severity is a property of the check rather than a setting: the number below is how many consecutive runs must see the new value before it becomes an email, and it is never less than two, because one observation is noise.

Every check is available on every plan, including the free one. CrawlWarden pricing, per site rather than per URL.

• the maintenance page that answers 200 instead
• the rest of what one robots.txt line can do
• an error served only to crawlers
• an expired certificate, the unplanned outage nobody schedules
• every failure mode CrawlWarden watches for
• every check it runs, and what a change in it means

What was checked, and where. An entry with no link is something that could not be verified, named here rather than left out.