How often a healthy site answers Googlebot differently: our measurement

The denominator is deliberately narrow. A site that is already broken for a browser tells you nothing about how it treats a crawler, so the 6 domains that answered nothing at all and the 20 that answered a browser with something other than 200 are excluded. What is left is 79 domains that were demonstrably healthy to an ordinary visitor at the moment they were probed.

5.1 percent is what the script measured and it is the figure recorded in the project's decision log, because the figure a test produces is the figure that gets written down. Reading the four cases by hand afterwards changes what they mean.

Two were genuine refusals: a 403 to the Googlebot user agent on a host that had just served a browser a 200. One was a 502 that occurred mid-redirect and ended on a different final URL, which has every appearance of a transient fault rather than a policy. One was not a block at all, but a 301 to the www. host that the probe recorded as a divergence.

So the genuine-refusal rate on this sample is 2.5 percent counting only the two 403s, and 3.8 percent if the 502 is counted as real. Anyone quoting this measurement should quote the range and say which end they are using. A page that reported only the largest of the three would be more quotable and less true.

Naming them is the point. An aggregate percentage with no cases behind it cannot be checked by anyone, and three of these four are large institutions whose behaviour is a matter of public record rather than anything private.

The script and the URL list are both in the repository this product is built from, so the measurement can be repeated by anyone rather than taken on trust.

• Date and place. 13 September 2026, from a single Hetzner host with German egress, address 178.105.60.3. One machine, one run.
• Population. 105 public domains extracted from a public corpus of r/SEO discussions, chosen because they are sites people were actually arguing about: real businesses, real WAFs, real CDNs, a mix of sizes and countries. No customer supplied any part of this list, and no customer site is in it.
• Per domain, two requests. curl -sL, following redirects, 20 second timeout, six domains in flight at once. The first sends a Chrome 127 user-agent string, the second sends Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html). Nothing else differs between them.
• Recorded per domain. The final HTTP status and the final URL after redirects, for each of the two fetches. The raw output is a TSV with one row per domain.
• Divergence, as defined here. The browser fetch ended on 200 and the Googlebot fetch did not. Nothing about page content is measured, and nothing about rendering.
• Volume. Two requests per domain, once. Less traffic than one human visit.

The reason a 5.1 percent figure is worth measuring at all is that none of these statuses is harmless to a page that is already indexed.

Google states that it "egresses primarily from IP addresses in the United States", and that if it detects a site blocking US requests "it may crawl from IP addresses in other countries" (Overview of Google crawlers and fetchers, updated 12 June 2026).

A geo rule written to block one region can therefore meet a crawler from that region. A probe that only ever asks from inside the United States cannot observe that case, which is why this measurement was taken from German egress and why the product's own check is.

The published figures in this area measure what sites declare. A census of the top 100,000 domains reports what share of their robots.txt files disallow a given agent, and those are useful numbers.

This one measures what a server does. A site can declare nothing at all in robots.txt and still answer a crawler-shaped request with a 403, because the decision was taken by a WAF, a bot-management product or a geo rule rather than by anyone editing a text file. The two measurements have different denominators and different causes and they are not comparable.

That distinction is the same one the product draws between its two Googlebot checks: one tests how a server behaves toward a crawler, the other reads what the site declares it allows.

None of these is a footnote. A reader deciding whether to cite the figure needs all six.

• Four events. On 4 divergences in 79 domains, the Wilson 95 percent interval runs from roughly 2.0 percent to 12.3 percent. The point estimate is not precise and should not be quoted as though it were.
• One run, one day, one address. Nothing here establishes that the figure is stable over time or from elsewhere. A WAF rule added the following week would change it.
• The population is skewed, in the direction that inflates the figure. All four divergent hosts are large publishers or institutions with aggressive bot protection. Nothing in the sample resembles a small site that wants to be crawled, which is the population the product mostly watches.
• This is not Googlebot. It is a datacentre client sending Googlebot's user-agent string. Google says that string is "often spoofed" and that requests should be verified by reverse DNS (Verifying Googlebot and other Google crawlers, updated 20 March 2026). A site that verifies properly would refuse this probe and admit the real crawler, so in that direction the figure reads high. A site allowlisting Google's published IP ranges would do the same. In the other direction nothing is measured at all.
• The measure is sensitive to how a final URL is compared. One of the four cases was a plain redirect to www., counted as divergence by a comparison that was too strict. That case is published rather than quietly removed, because it is the clearest available illustration of how a divergence figure can be inflated by its own definition.
• Nothing here is a statement about intent. Google defines cloaking as different content served to users and search engines with the intent to manipulate rankings and mislead users (Spam policies for Google web search). A 403 from a firewall is a crawl failure, costs the same traffic, and needs an entirely different remedy. This measurement observes status codes and cannot see why any of them was sent.

Two files do the whole job: a shell script of about sixty lines, and a text file of 105 domains. Copy both to a host with non-US egress, make the script executable, and pass it the list. It prints the counts, the divergence percentage and the divergent hosts, and writes the full TSV beside itself. Set CONCURRENCY=2 if 429s appear, because at that point the rate limiting is yours rather than theirs.

The gate it was built to pass was written down before the number was known, which is the only order in which a threshold means anything.

5.1 percent landed in the middle band, so the check shipped with two conditions attached rather than unconditionally.

Egress sits behind an interface, so a different route can be swapped in without touching the checking engine. And a host that repeatedly refuses the Googlebot profile is marked unverifiable rather than reported as a change: the customer is told once, and that check stops alerting on that host. A false "Googlebot cannot reach this page" is worse than no alert at all, because the check it discredits is the one nothing else does.

The figure is also the reason the product will re-measure against real customer properties once there are enough of them to be worth measuring. Everything on this page is a measurement of other people's public sites, and it is stated as such.

The sentence that carries all of it: of 79 public domains that returned 200 to a browser on 13 September 2026, 4 (5.1 percent, Wilson 95 percent interval 2.0 to 12.3 percent) returned a non-200 status to a client sending Googlebot's user-agent string from a German datacentre address, of which 2 were 403 refusals. Measured by CrawlWarden.

Corrections are welcome and will be dated on this page. If the method looks wrong, it is written out above in enough detail to say where.

• Google: Overview of Google crawlers and fetchers, updated 12 June 2026. That Google egresses primarily from US addresses and may crawl from other countries when a site blocks US requests.
• Google: HTTP status codes and network errors, updated 4 February 2026. The handling of 403, 502 and 301, and the 10 hop redirect limit.
• Google: Verifying Googlebot and other Google crawlers, updated 20 March 2026. That the user-agent string is often spoofed and verification is by reverse DNS.
• Google: Spam policies for Google web search. That the definition of cloaking includes intent, which is why nothing here is called cloaking.
• Not verified: whether the 5.1 percent figure generalises beyond this sample. It is one run of 79 domains on one day from one address, and no comparable published measurement of server behaviour toward a Googlebot user agent was found.
• Not verified: whether the two 403 hosts verify crawlers by reverse DNS and would have admitted the real Googlebot. Determining that from outside is not possible, so both remain counted as refusals of this probe rather than of Google.

• what to do when a site answers crawlers differently from browsers
• the twelve indexability checks, including the two Googlebot ones
• the Cloudflare settings that produce a 403 to Googlebot
• what the CrawlWarden crawler does, and the rules it holds itself to
• every check CrawlWarden runs, by group