Indexing & AI Crawler Checkers
A page can only rank, or be cited by an AI assistant, if crawlers can reach it and are allowed to index it. robots.txt, noindex directives, HTTP status codes, redirects and sitemaps decide that.
These checkers test those signals for one URL. The AI crawler checks report what your robots.txt allows. They are readiness signals, not a guarantee that any AI product will use your content.
AI Crawler Blocked Checker
Reads robots.txt and reports whether GPTBot, ClaudeBot, PerplexityBot and other AI crawlers are allowed in.
Crawlability Test
Checks status code, robots.txt, and noindex tag/header together — everything that decides whether a crawler can index the page.
llms.txt Checker
Checks whether the site publishes an llms.txt file at its root and, if it does, that it has a title and markdown links.
Googlebot Simulator
Strips the page to title, headings, readable text and links — what a basic crawler extracts.
Noindex Checker
Checks the meta robots tag and the X-Robots-Tag header for noindex, and says whether the page can be indexed.
Robots.txt Checker
Checks robots.txt exists, returns 200, and does not block this page.
URL Status Code & Redirect Checker
Reports the HTTP status code and the full redirect chain for the URL, however the redirects are configured.
XML Sitemap Checker
Checks for a valid XML sitemap, declared in robots.txt or at /sitemap.xml.