Cloudflare is answering Googlebot with 403 - how do I confirm it and fix it?

Through mid-2026 this was a real and widely reported problem. Cloudflare classifies Googlebot as a mixed-purpose crawler, because the same bot feeds both Search and Google AI products, and the blanket block settings applied to it. Cloudflare said so in the open: multi-purpose crawlers "such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training" (Cloudflare, Your site, your rules, 1 July 2026).

That changed on 15 September 2026. Cloudflare now labels Googlebot, Bingbot and Applebot "Accountable", and under the Disallow AI Training setting "Accountable mixed-use crawlers remain allowed for search" (Cloudflare, Have it both ways). Earning that label takes four commitments from the operator: an opt-out from AI training through robots.txt or an equivalent, an opt-out from AI summaries, URL-level visibility into which pages were made available for training, and an assurance that opting out does not affect search rankings.

The legacy Block AI bots toggle was deprecated on the same date, and its own documentation already said it "excludes mixed-purpose bots that are used both for Training and for Search" (Cloudflare bot docs). Advice written between July and September 2026 saying that blocking AI training kills your Google traffic was true when it was written and is not true now.

In the threads where someone actually gets to the bottom of a Googlebot 403, the cause is a security rule nobody connected to search. A Cloudflare Community thread from 4 November 2025 shows how specific it gets. A phpBB forum owner had three WAF Skip rules for Googlebot, Bingbot and APIs-Google at the top of the list, could see them matching, had turned off Bot Fight Mode, Block AI Bots and Browser Integrity Check, and was still getting "Blocked due to access forbidden (403)" in Search Console. A community MVP told them to stop guessing and read the Security Events log. The answer was a typo: "In one of our rules we authorised 'googleblot' and it seems we should have authorized 'google'" (Cloudflare Community).

• A custom WAF rule with a misspelled user agent, or one that matches user agent and country together.
• Bot Fight Mode or Super Bot Fight Mode, in a configuration where verified bots are not exempt from everything else.
• A managed ruleset firing on something incidental - an oversized cookie, an unusual header - rather than on the bot at all.
• A country block. Googlebot crawls from several countries, and a rule that allows only your home market will refuse it.
• Rate limiting with a 403. Google documents this as the wrong tool: "Don't use 401 and 403 status codes for limiting the crawl rate."
• An origin firewall. Cloudflare gets blamed for a fair number of 403s it did not serve.

• In Cloudflare, open Security then Events for the zone and filter for the blocked requests. The log names the service that acted. This is the step that turns the exercise from guesswork into a one-line fix, and it is the step people skip.
• Check the client really was Googlebot and not something wearing its user agent. Google documents the reverse DNS method: resolve the IP, confirm the hostname ends in googlebot.com, google.com or googleusercontent.com, then resolve that hostname forward and confirm it returns the original IP (Verifying Googlebot).
• For a rule rather than a one-off check, Google publishes the ranges in CIDR form as JSON, including common-crawlers.json for Googlebot and special-crawlers.json for the rest.
• In Search Console, run URL Inspection then Test Live URL. A 403 there, on a page your browser loads, is the edge and nothing else.
• From a shell: curl -s -o /dev/null -w "%{http_code}" -A "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" https://example.com/. A 403 here proves a user-agent rule. A 200 proves nothing, because most rules weigh the source IP as well.
• As a last resort, set the record to DNS-only for a few minutes. If the 403 goes away it is Cloudflare; if it does not, it is your origin.

Google is unambiguous about 4xx. It "doesn't index URLs that return a 4xx status code, and URLs that are already indexed and return a 4xx status code are removed from the index" (HTTP status codes and network errors). Google publishes no figure for how many consecutive 403s it takes, and the honest answer is that it depends on how often the URL is crawled: a page Googlebot visits daily leaves sooner than one it visits monthly.

If you genuinely need a crawler to slow down, 503 is the correct answer rather than 403. The same reference says 5xx and 429 "prompt Google's crawlers to temporarily slow down with crawling", and that already-indexed URLs are "preserved in the index, but eventually dropped" rather than removed on contact.

One widely shared figure claims a merchant lost 64 percent of impressions after Cloudflare AI defaults blocked Googlebot. It comes from a single agency blog post about a single store, with none of the underlying data published. It is plausible and it is not evidence, so this page treats it as an anecdote and nothing is argued from it.

• Read Security Events and identify the service that blocked the request. Change nothing before this step; a fix applied at the wrong layer leaves a hole open and does not solve the problem.
• If a custom rule did it, fix that rule rather than stacking an exception above it, and re-read every user agent string in it for spelling.
• If a managed ruleset did it, add a skip scoped to the specific rule ID for verified bots, not a blanket skip of the ruleset.
• If the bots setting did it, move from Block to Disallow AI Training. That keeps Googlebot, Bingbot and Applebot crawling for search while still refusing the training-only crawlers.
• Re-test with URL Inspection, then watch the 403 count in Search Console fall over the following days. None of this is instant.

The site is up. Analytics looks normal, because human visitors were never blocked. Uptime monitoring is green, because the monitor is not Googlebot and is usually on an allowlist. The only two places this surfaces are a log nobody reads daily and a Search Console report that lags by days.

The general shape of it is a site answering a crawler differently from a browser, which is a problem older than every AI crawler argument. What is new is that one toggle in one dashboard can cause it across an entire zone, and that the same edge can rewrite the robots.txt a crawler reads without anyone deploying anything.

It is also not a rare shape. CrawlWarden fetched 105 public domains twice, as a browser and as Googlebot, and of the 79 that were healthy to the browser, 4 answered the crawler with something else: two of them a flat 403 (the measurement in full, with its limits).

CrawlWarden watches a list of URLs you choose and reports what changed. It does not diagnose and it does not score. Severity is a property of the check rather than a setting: the number below is how many consecutive runs must see the new value before it becomes an email, and it is never less than two, because one observation is noise.

Every check is available on every plan, including the free one. CrawlWarden pricing, per site rather than per URL.

• the general case of a site answering crawlers differently
• robots.txt rules that block Googlebot outright
• the AI crawlers on the other side of the same setting
• the 503 you serve on purpose, and the one you do not
• every failure mode CrawlWarden watches for
• every check it runs, and what a change in it means

What was checked, and where. An entry with no link is something that could not be verified, named here rather than left out.