The CrawlWarden crawler

CrawlWarden fetches a small, customer-curated list of pages every few hours to detect changes that affect whether search engines can index them. Almost every page it requests was entered by a customer who asked for it to be monitored.

The exception is the free X-Robots-Tag checker, where a visitor pastes one URL and we fetch that URL once, on the spot, to show what it returned. Those requests carry the same user agent and are described below.

User-agent: CrawlWarden/1.0 (+https://crawlwarden.com/bot)

• Requests only the specific URLs a customer listed. It does not follow links and it does not crawl a site.
• Requests each page a second time with a Googlebot user agent, to detect whether a site answers search crawlers differently from browsers. This is done on customers' own declared properties, at their request, and never to bypass anything that refused our own user agent.
• Fetches one URL, once, when a visitor pastes it into the free checker. That request follows at most five redirects, reads at most the first megabyte, gives up after five seconds, and is rate limited per visitor. It uses our own user agent and never Googlebot's, it never requests anything else on the host, and it is sent from our Vercel infrastructure rather than from the address the monitoring crawler uses.
• Respects Retry-After.
• Fetches at most once every three hours per page, and once a day on the free plan.

Add this to your robots.txt:

User-agent: CrawlWarden
Disallow: /

Or email support@crawlwarden.com and we will stop fetching your host. A request from a site owner is acted on the same day.

CrawlWarden is not a proxy or VPN service. It makes outbound requests to properties its customers have declared, in order to report on them, and to a single URL at a time in the free checker, in order to show the visitor what that URL returned. It does not relay third-party traffic, mask anyone's origin, or provide network access to anyone. The checker returns a fixed set of fields about one response and never the page itself, it refuses addresses that are not on the public internet, and it does not follow links.