Robots.txt Checker

robots.txt is a plain-text file at the root of a site (example.com/robots.txt) that tells crawlers which URLs they may request. It controls crawling, not indexing: a blocked URL can still appear in Google if other pages link to it. Search engines and most AI crawlers follow it, but it is a request, not access control.

1. Enter any URL on the site. The tool fetches /robots.txt from that host.
2. It checks that the file exists, returns HTTP 200, and does not block the URL you entered for Googlebot.
3. If the page is blocked, the result shows which rule matched so you can edit or remove it.

Passed

robots.txt returns 200 and allows Googlebot to crawl the URL you entered.

Needs attention

robots.txt is missing (404). Google then treats the whole site as crawlable, which is usually fine, but you lose the place to list your sitemap.

Failed

A Disallow rule blocks the URL you entered, so Google cannot crawl it or see its content.

Start by allowing everything and add Disallow lines only for areas with no search value, such as internal search results, carts or account pages. Rules are prefix matches and are case-sensitive. List your sitemap with an absolute URL.

User-agent: *
Disallow: /cart/
Disallow: /search

Sitemap: https://www.example.com/sitemap.xml

import type { MetadataRoute } from 'next';

export default function robots(): MetadataRoute.Robots {
  return {
    rules: { userAgent: '*', disallow: ['/cart/', '/search'] },
    sitemap: 'https://www.example.com/sitemap.xml',
  };
}

Use a noindex robots meta tag or an X-Robots-Tag: noindex header, and leave the page crawlable. Google has to crawl a page to see its noindex, so blocking it in robots.txt as well stops the noindex from working.

A leftover "Disallow: /" from a staging site blocks everything. Blocking CSS or JavaScript files stops Google rendering the page properly. Putting the file anywhere other than the root of the host means it is ignored, and each subdomain and protocol needs its own file.

• Introduction to robots.txt (Google Search Central)
• How Google interprets the robots.txt specification
• RFC 9309: Robots Exclusion Protocol
• Block Search indexing with noindex (Google Search Central)

• robots.txt is present and usable

  Checks that /robots.txt exists, returns 200, and does not block the audited page for Googlebot.

  Why it matters: robots.txt controls which URLs crawlers may request. A missing file is usually harmless, but an over-broad Disallow can hide important pages from search engines.

  If it fails: Publish a robots.txt at the site root. Make sure it does not Disallow pages you want indexed, and reference your sitemap.

  Source: developers.google.com/search/docs/crawling-indexing/robots/intro

• AI Crawler Blocked CheckerReads robots.txt and reports whether GPTBot, ClaudeBot, PerplexityBot and other AI crawlers are allowed in.
• Crawlability TestChecks status code, robots.txt, and noindex tag/header together — everything that decides whether a crawler can index the page.
• llms.txt CheckerChecks whether the site publishes an llms.txt file at its root and, if it does, that it has a title and markdown links.
• Googlebot SimulatorStrips the page to title, headings, readable text and links — what a basic crawler extracts.
• Noindex CheckerChecks the meta robots tag and the X-Robots-Tag header for noindex, and says whether the page can be indexed.
• URL Status Code & Redirect CheckerReports the HTTP status code and the full redirect chain for the URL, however the redirects are configured.