3 min read
Is my robots.txt blocking Googlebot?
By CrawlWarden, which builds the monitoring described at the foot of this article.
Published , sources last re-read .
Check for Disallow: / under a User-agent: * or User-agent: Googlebot group at https://yourdomain.com/robots.txt. That single line blocks crawling of the entire site. Be aware of what it does not do: Disallow blocks fetching, not indexing. A blocked URL can still appear in search results from inbound links alone, showing no description - and because Google may not fetch the page, it can never read a noindex tag you put there to remove it.
How a crawler picks which group applies
A crawler obeys exactly one group: the one whose User-agent most specifically matches its own name. If a User-agent: Googlebot group exists, Googlebot reads that group and completely ignores the User-agent: * group, including any rules only present there.
This is the single most common robots.txt misreading. Adding a narrow Googlebot group to grant one exception silently drops every other rule you had written for everyone.
Within a group, the most specific rule wins
Google resolves conflicting Allow and Disallow rules by path length, not by order in the file. Allow: /data/reports/ beats Disallow: /data/ because it is longer. When two rules are the same length, Allow wins.
The AI crawlers are a separate question
GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended each read robots.txt under their own names. A site can be perfectly open to Googlebot and invisible to every assistant, and the owner will see nothing in Search Console about it.
Whether to allow them is a genuine business decision, not an oversight - some publishers block them deliberately. The failure worth catching is blocking them by accident, or believing you allow them when a bot-protection layer is returning 403 regardless of what the file says.
- Bot-protection defaults are the usual culprit. Cloudflare’s Bot Fighting Mode has flagged crawlers marked exempt, and a fake-Googlebot flood that triggers 503s does the damage at exactly the wrong moment.
- Test with the user agent, not just the file:
curl -A "GPTBot" -sI https://example.com/page.
Disallow, noindex and 410, one more time
| You want to... | Use | Not |
|---|---|---|
| Remove a page from search | noindex, and let it be crawled | Disallow - Google cannot read the tag |
| Stop crawl waste on faceted URLs | Disallow | noindex - it does not stop crawling |
| Remove thousands of pages permanently | HTTP 410 | noindex - needs a re-fetch of every URL |
The checks that report this
CrawlWarden watches a list of URLs you choose and reports what changed. It does not diagnose and it does not score. Severity is a property of the check rather than a setting: the number below is how many consecutive runs must see the new value before it becomes an email, and it is never less than two, because one observation is noise.
robots.txt disallowconfirmed over 2 consecutive runs
Evaluates the live robots.txt against each watched URL, for each crawler you care about, and reports when one becomes blocked.
robots.txt reachableconfirmed over 2 consecutive runs
Reports when robots.txt stops being fetchable at all.
robots.txt contentconfirmed over 2 consecutive runs
Reports any change to the file’s contents, whether or not it changes a verdict for a watched URL.
Every check is available on every plan, including the free one. CrawlWarden pricing, per site rather than per URL.
Questions
- Does Disallow remove a page from Google?
- No. It stops Google fetching the page. The URL can still be listed from inbound links, typically with no snippet, and Google cannot see a noindex tag on a page it is not allowed to fetch.
- Is a missing robots.txt a problem?
- No. A 404 for robots.txt means "crawl everything", which is a valid state. What causes real trouble is a robots.txt that returns a 5xx: several crawlers treat a server error as "do not crawl anything" until it clears.
- Should I allow GPTBot and ClaudeBot?
- That is a business call about whether you want to be quotable in assistant answers. Both positions are defensible. What is not defensible is not knowing which one your robots.txt currently takes.
Related
- a site that serves crawlers different content
- noindex, which does the job Disallow is often mistaken for
- whether the AI crawlers can reach you, which robots.txt no longer answers
- the staging Disallow that reaches production
- every failure mode CrawlWarden watches for
- every check it runs, and what a change in it means
Sources
What was checked, and where. An entry with no link is something that could not be verified, named here rather than left out.
Google: How Google interprets the robots.txt specification
Both matching sections: that a crawler takes "the group with the most specific user agent that matches" and does not merge groups, and that within a group the longest matching rule path wins, with "the least restrictive rule" breaking an allow-versus-disallow tie.
Google Search Central: Introduction to robots.txt
That Disallow governs crawling rather than indexing, and that a disallowed URL can still appear in results without a description.
Google Search Central: Block search indexing with noindex
The trap in the last section: a page blocked in robots.txt is never fetched, so a noindex placed on it is never read.
Not verified: search volume for this query
No keyword volume tool is connected to this project. Demand was judged from a live depersonalised US SERP.
Know the moment this happens to you
Give CrawlWarden a domain, keep the pages it suggests, and the first check runs immediately. Free forever for one site.