3 min read

My site answers Googlebot differently from a browser - is that cloaking?

By CrawlWarden, which builds the monitoring described at the foot of this article.

Published , sources last re-read .

Usually not, and the traffic goes anyway. Google defines cloaking as presenting different content to users and search engines "with the intent to manipulate search rankings and mislead users", so a firewall rule that returns 403 to Googlebot by accident is not a spam violation. It is a crawl failure, and it costs the same pages. The distinction only changes the remedy: deliberate cloaking needs a reconsideration request, an accidental 403 needs the rule fixed.

The accidental versions

  • A WAF or bot-management rule rate-limits or challenges a crawler. A JavaScript challenge is an empty page to anything that does not run JavaScript.
  • Geo-blocking. Googlebot crawls predominantly from US addresses; a site that blocks or redirects non-domestic traffic answers Google from a country its owners never tested.
  • A CDN serving a cached variant negotiated for one user agent to every user agent.
  • A paywall or consent wall that returns the article to browsers and a stub to everything else.
  • Server-side user-agent detection meant to serve a lighter page to bots. Google says it generally does not treat this as cloaking while the content stays similar, so the risk is that "lighter" quietly becomes "different" and nobody re-reads the two side by side.

Why it is so hard to see

Every check you run from your own browser passes. The site is fine from your desk, fine from your phone, fine on the staging domain. The divergence exists only for a request that carries a crawler user agent, and often only from a particular address range.

It is also commoner than it looks. CrawlWarden probed 105 public domains twice, as a browser and as Googlebot, and 5.1 percent of the ones that were healthy to a browser answered the crawler with something else (the full measurement, method and limits).

Search Console’s URL Inspection "Test Live URL" is the closest first-party tool, and it is a manual, one-URL-at-a-time check that nobody runs on a schedule.

How to reproduce it yourself

Fetch the same URL twice, changing only the user agent, and compare the status and the body:

  • curl -sI https://example.com/page - a normal request.
  • curl -sI -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/page - the same request as a crawler.
  • Different status codes, different content lengths, or a challenge page in one and not the other is the answer. Repeat from a non-US address if you can; geo rules will not show up otherwise.
  • Verify a real Googlebot in your own logs by reverse DNS on the IP - googlebot.com or google.com - then forward-resolve it. Fake Googlebot traffic is common and is a different problem.

This is not the same as dynamic rendering

Serving prerendered HTML to everyone, crawlers and browsers alike, is not cloaking - it is just server-side rendering, and it is the right fix for a client-rendered site that Googlebot sees as an empty shell. The line is whether the decision depends on who is asking.

The checks that report this

CrawlWarden watches a list of URLs you choose and reports what changed. It does not diagnose and it does not score. Severity is a property of the check rather than a setting: the number below is how many consecutive runs must see the new value before it becomes an email, and it is never less than two, because one observation is noise.

  • Bot divergenceconfirmed over 2 consecutive runs

    Fetches each watched page twice - once as a browser, once as Googlebot - and reports when the two answers stop agreeing.

  • Googlebot statusconfirmed over 2 consecutive runs

    Reports the status Googlebot receives, fetched from a non-US address so geo rules are visible.

Every check is available on every plan, including the free one. CrawlWarden pricing, per site rather than per URL.

Questions

Is prerendering my React app cloaking?
No, as long as every visitor gets the prerendered HTML. Cloaking is about serving different content based on who is asking. Serving the same server-rendered page to everyone is simply how the web worked before single-page apps.
What if the content is the same and only the markup differs?
Google says it generally does not treat this as cloaking while the content stays similar, because the definition turns on intent to manipulate rankings and mislead users. The practical risk is different: "lighter" quietly becomes "different" over a few releases and nobody re-reads the two side by side. Compare them on a schedule rather than relying on the definition.
Why fetch from a non-US address?
Because a geo-block that catches Googlebot is invisible from inside the country the site serves. A US-only check would pass on exactly the sites that have the problem.

Related

Sources

What was checked, and where. An entry with no link is something that could not be verified, named here rather than left out.

  • Google Search Central: Spam policies for Google web search

    The definition the answer block turns on: cloaking is "presenting different content to users and search engines with the intent to manipulate search rankings and mislead users". Intent is part of the definition, which is why an accidental 403 is a crawl failure rather than a violation.

  • Google Search Central: Dynamic rendering as a workaround

    That "Googlebot generally doesn’t consider dynamic rendering as cloaking" while the content stays similar, and that Google calls it "a workaround and not a recommended solution".

  • Google Search Central: Verifying Googlebot and other Google crawlers

    How to confirm a request really came from Googlebot before concluding anything from a user agent, behind the reproduce-it-yourself section.

  • Not verified: that almost every real case is accidental

    This matches what the failure looks like in support and in the incident write-ups behind this page. Nobody publishes a split between deliberate and accidental cases.

Know the moment this happens to you

Give CrawlWarden a domain, keep the pages it suggests, and the first check runs immediately. Free forever for one site.