AI Crawler Blocked Checker
Reads robots.txt and reports whether GPTBot, ClaudeBot, PerplexityBot and other AI crawlers are allowed in.
How do I check if my site is blocking AI crawlers?
Enter your URL and this tool reads your robots.txt, then reports which AI crawlers it allows and which it blocks, including OpenAI’s GPTBot and OAI-SearchBot, Anthropic’s ClaudeBot and Claude-SearchBot, PerplexityBot, Google-Extended and Common Crawl’s CCBot. AI companies run separate bots for model training, for search indexes and for fetching pages a user asks about.
How to use the AI Crawler Blocked Checker
- Enter any URL on the site. The tool fetches its robots.txt.
- It checks the rules that apply to each AI crawler’s user-agent token.
- Review the allowed and blocked lists and decide whether they match your policy.
How to read the results
- Informational
- The result lists allowed and blocked AI crawlers. It is informational and does not change your score, because blocking AI crawlers is a legitimate choice.
Which AI crawlers should I allow?
Decide separately for search and for training. Search and user-triggered bots (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot) fetch pages so AI assistants can cite and link to you. Training bots (GPTBot, ClaudeBot, CCBot) collect content to train models. Google-Extended and Applebot-Extended are not crawlers: they are tokens that control whether content Googlebot or Applebot already fetched can be used for Gemini or Apple’s models. Blocking Google-Extended does not affect Google Search.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: *
Allow: /robots.txt allows the bot but it still cannot reach my site
Many CDNs and firewalls block AI bots separately from robots.txt. Cloudflare, for example, has an AI crawler setting that can return 403 to bots your robots.txt allows. Check your CDN or WAF bot settings as well as the file.
Frequently asked questions
Does blocking GPTBot remove my site from ChatGPT search?
No. OpenAI documents GPTBot as its training crawler. ChatGPT search uses OAI-SearchBot, and ChatGPT-User fetches pages users ask about. Block or allow them separately.
Do AI crawlers obey robots.txt?
The major operators (OpenAI, Anthropic, Google, Apple, Perplexity, Common Crawl) say their crawlers follow it. Some operators say user-triggered fetchers may not apply robots.txt rules, because a person asked for that page. robots.txt is a convention, not access control.
Will allowing AI crawlers get me cited by AI assistants?
It makes citation possible; it does not guarantee it. AI assistants choose sources based on relevance, quality and their own ranking systems.
Sources
What this tool checks
AI crawler access in robots.txt
Reports whether robots.txt blocks well-known AI crawlers (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, CCBot, …).
Why it matters: Blocking AI crawlers is a legitimate choice, but if you want your content to be usable by AI assistants they must be allowed to fetch it. This is an optimization/readiness signal only — it does not guarantee visibility in ChatGPT, Google AI Overviews or any other AI system.
If it fails: Decide deliberately: allow the AI crawlers you are comfortable with, and block the rest in robots.txt.
Source: platform.openai.com/docs/bots
Related tools
- Crawlability TestChecks status code, robots.txt, and noindex tag/header together — everything that decides whether a crawler can index the page.
- llms.txt CheckerChecks whether the site publishes an llms.txt file at its root and, if it does, that it has a title and markdown links.
- Googlebot SimulatorStrips the page to title, headings, readable text and links — what a basic crawler extracts.
- Noindex CheckerChecks the meta robots tag and the X-Robots-Tag header for noindex, and says whether the page can be indexed.
- Robots.txt CheckerChecks robots.txt exists, returns 200, and does not block this page.
- URL Status Code & Redirect CheckerReports the HTTP status code and the full redirect chain for the URL, however the redirects are configured.
Reviewed by Prabath Thalangama · Last updated