Is my site blocked from ChatGPT, Claude and the other AI crawlers?

Almost every argument about this conflates three separate things. A crawler can be collecting content to train a model, indexing content so it can be cited in an answer, or fetching a page right now because a user asked something. Vendors ship different bots for each, and you can allow one and refuse another.

These are the current tokens, from each vendor’s own documentation:

Cloudflare replaced its single "block AI bots" switch with three categories, Search, Training and Agent, each carrying its own setting. Search and Agent can be set to Allow, Block, or Block on pages with ads. Training adds a fourth, Disallow AI Training, which publishes the no-training preference in your robots.txt instead of refusing the request.

From 15 September 2026, a new domain with no ads defaults to Allow on all three. A new domain that displays ads defaults to Allow for Search, Disallow AI Training for Training, and Block on pages with ads for Agent. Existing customers had to record a preference before that date to avoid the change. Note which setting the ad-bearing default picks: it is the one that keeps search working.

The important part for anyone trying to diagnose this: that block is applied at the edge. Your robots.txt is untouched and still says whatever it always said. A crawler is refused before it ever reaches your server, and the file you would check to find out is not the file doing it.

The same is true of any WAF rule, bot-management setting or rate limit anywhere in front of your origin. If your site sits behind a CDN, robots.txt describes your intent and the edge decides the outcome.

This is new, and it is the part worth reading twice. Until 15 September 2026 the Block settings deliberately spared the mixed-use crawlers. Cloudflare’s own wording: Block and "Block on pages with ads" "previously did not apply to mixed-use crawlers because blocking them could also affect search discoverability". That exception is gone. Block and Block on pages with ads now apply to every training crawler, Googlebot, Bingbot and Applebot included.

The default is not the dangerous setting, and the distinction is the whole point. Disallow AI Training exists precisely so this trade does not have to be made: Cloudflare designates Google, Apple and Microsoft as Accountable, and that setting leaves their crawlers allowed for search while refusing training. Amazon, Anthropic, Meta and OpenAI are Accountable for a different reason, which is that they ship separate search and training crawlers in the first place.

The risk is in choosing Block, which is a different control and a deliberate act. Someone ticks a box meaning "do not train on my content", and the consequence is that Googlebot stops being able to fetch the site. Nothing about the site looks different, the setting was chosen for a defensible reason, and the person who ticked it is often not the person who watches search traffic.

Note the separate case that is safe: Google’s own Google-Extended token is training-only, and Google states it "does not impact a site’s inclusion in Google Search nor is it used as a ranking signal". Blocking Google-Extended in robots.txt is not the same as blocking Training at a CDN, and does not carry the same risk.

Fetch your own page as each crawler and read the status. This is the only method that sees an edge block, because it is the only one that actually asks:

• curl -sI -A "Mozilla/5.0 (compatible; GPTBot/1.4; +https://openai.com/gptbot)" https://example.com/page
• Repeat for ClaudeBot, OAI-SearchBot, PerplexityBot and Googlebot. A 200 for one and a 403 or a challenge for another is your answer.
• Compare against a plain browser fetch of the same URL. If they differ, something between you and the visitor is making a decision you did not make on the page.
• Verify a crawler in your own logs before trusting the user agent: OpenAI publishes IP ranges at openai.com/gptbot.json and one file per bot, Anthropic at claude.com/crawling/bots.json. Anyone can send a user agent string; only the real crawler comes from those ranges.
• If you block by IP, note Anthropic’s own warning that doing so stops them fetching your robots.txt at all, so you lose the ability to express anything more nuanced later.

All of the above assumes a crawler says who it is. Not all of them do, and this is where honest advice runs out.

In August 2025 Cloudflare published findings that Perplexity, when its declared crawler hit a block, switched to a generic browser user agent impersonating Chrome on macOS and rotated source networks, generating millions of daily requests across tens of thousands of domains. Cloudflare tested it on freshly created domains whose robots.txt forbade all automated access, and said the content still came back in answers. It de-listed Perplexity as a verified bot in response. Perplexity disputed the characterisation.

Take from that what it actually supports: robots.txt is a request, not a fence, and it only binds the crawlers that choose to read it. A site that wants a hard boundary needs one enforced at the network, and a site that wants to be cited needs to confirm the crawlers can actually get in rather than assuming the file did its job.

It is the first thing suggested whenever this comes up, and the evidence against it is unusually clear. Ahrefs analysed 137,000 domains: 28% publish an llms.txt, 97% of those files received no traffic in a month, 96% of the fetches that did happen were bots, and no request ever arrived for an llms.txt that did not exist. Nothing goes looking for it.

John Mueller’s position is to make one when a platform that actually sends you customers says it needs the file. There is a narrower nuance worth knowing: Google’s Search guidance says you do not need it for generative search visibility, while Chrome’s agentic browsing documentation lists it as a signal for agents navigating a site to complete a task. Those are different surfaces and only one of them is about being cited.

If the concern is whether models can reach and quote you, an llms.txt answers none of it. Access does.

CrawlWarden watches a list of URLs you choose and reports what changed. It does not diagnose and it does not score. Severity is a property of the check rather than a setting: the number below is how many consecutive runs must see the new value before it becomes an email, and it is never less than two, because one observation is noise.

Every check is available on every plan, including the free one. CrawlWarden pricing, per site rather than per URL.

• how to read a robots.txt rule the way a crawler does
• a site answering crawlers differently from browsers
• noindex, the other way a page silently leaves search
• every failure mode CrawlWarden watches for
• every check it runs, and what a change in it means

What was checked, and where. An entry with no link is something that could not be verified, named here rather than left out.