Robots.txt Analyzer
Fetch and parse any site's robots.txt: crawl rules, disallowed paths, per-bot directives, crawl delays, and sitemap references. Validate that search engines can crawl what you intend.
Fetching robots.txt, please wait...
About robots.txt and this analyzer
The robots.txt file lives at the root of a website and tells web crawlers which parts of the site they may or may not fetch. This analyzer downloads that file and breaks it down per user-agent, showing every Allow and Disallow rule, any Crawl-delay directive, and the sitemap URLs the site declares.
A common misconception is that Disallow hides a page from search engines. It does not — robots.txt only controls crawling, not indexing. A disallowed URL can still appear in search results if other pages link to it; to keep a page out of the index you need a noindex meta tag or an X-Robots-Tag header, which requires the page to stay crawlable.
Use this tool to confirm that important pages are not accidentally blocked, that your sitemap is referenced, and that crawl rules behave the same for Googlebot as for other crawlers.
Frequently asked questions
What does a robots.txt checker do?
It fetches a site's /robots.txt file and parses it, showing which user-agents are addressed, which paths are Allowed or Disallowed, any crawl-delay directives, and the sitemap URLs it declares.
How do I check if a page is blocked by robots.txt?
Enter the site's domain above. The analyzer lists every Disallow rule per user-agent, so you can see whether the path you care about is blocked for Googlebot or another crawler.
Does blocking a page in robots.txt remove it from Google?
No. robots.txt only asks crawlers not to fetch a URL; a blocked URL can still be indexed if other pages link to it. To keep a page out of search, use a noindex meta tag or an X-Robots-Tag header instead.