Crawlability Test
Checks status code, robots.txt, and noindex tag/header together — everything that decides whether a crawler can index the page.
What this tool checks
Page returns a 200 status
Checks the final response status of the audited page is 200 OK.
Why it matters: A page that returns an error or unexpected status will not be indexed and signals a broken experience to users.
If it fails: Return HTTP 200 for pages that should be live and indexable.
Source: developers.google.com/search/docs/crawling-indexing/http-network-errors
robots.txt is present and usable
Checks that /robots.txt exists, returns 200, and does not block the audited page for Googlebot.
Why it matters: robots.txt controls which URLs crawlers may request. A missing file is usually harmless, but an over-broad Disallow can hide important pages from search engines.
If it fails: Publish a robots.txt at the site root. Make sure it does not Disallow pages you want indexed, and reference your sitemap.
Source: developers.google.com/search/docs/crawling-indexing/robots/intro
Robots meta directives
Reports the directives found in <meta name="robots"> (and googlebot).
Why it matters: Robots meta directives such as noindex or nofollow directly control whether a page is indexed and whether its links are followed.
If it fails: Only use noindex/nofollow where you truly want to exclude the page or its links.
Source: developers.google.com/search/docs/crawling-indexing/robots-meta-tag
X-Robots-Tag header
Reports the directives found in the X-Robots-Tag HTTP response header.
Why it matters: X-Robots-Tag applies robots directives at the HTTP level and can silently deindex a page even when the HTML looks fine.
If it fails: Check that any X-Robots-Tag noindex/nofollow is intentional.
Source: developers.google.com/search/docs/crawling-indexing/robots-meta-tag
Related tools
- AI Crawler Blocked CheckerReads robots.txt and reports whether GPTBot, ClaudeBot, PerplexityBot and other AI crawlers are allowed in.
- llms.txt CheckerChecks whether the site publishes an llms.txt file at its root and, if it does, that it has a title and markdown links.
- Googlebot SimulatorStrips the page to title, headings, readable text and links — what a basic crawler extracts.
- Noindex CheckerChecks the meta robots tag and the X-Robots-Tag header for noindex, and says whether the page can be indexed.
- Robots.txt CheckerChecks robots.txt exists, returns 200, and does not block this page.
- URL Status Code & Redirect CheckerReports the HTTP status code and the full redirect chain for the URL, however the redirects are configured.