See Exactly What Your Robots.txt Blocks or Allows

What a robots.txt file controls

A robots.txt file sits at the root of a domain and tells crawlers which parts of a site they're allowed to request. It's a set of instructions: a well-behaved crawler like Googlebot will follow the rules, but nothing stops a scraper from ignoring the file entirely. That distinction matters when you're troubleshooting why a page still turned up in search results despite being disallowed, because a disallow rule stops crawling.

Why testing the file matters before you publish it

A single misplaced wildcard, or an Allow rule written after a broader Disallow, can block far more of a site than intended. The mistake often goes unnoticed until crawl budget (the number of pages a search engine is willing to crawl in a given period) starts dropping on pages that were never meant to be hidden. Testing the file against real paths before it goes live catches that kind of error while it's still cheap to fix.

What this tool checks

• Whether a given URL path is allowed or disallowed for a chosen user-agent
• Which specific rule in the file is the one being matched
• Conflicts between Allow and Disallow lines that apply to the same path
• Basic syntax problems, such as a missing user-agent line before a set of rules

Paste in a live robots.txt file, or write one from scratch, and test it against the URLs you actually care about. The result reflects how the rules parse, not a live crawl of Google's index, so treat it as a check on the file's logic rather than confirmation of how any particular search engine is behaving on your site right now.

Where robots.txt rules commonly go wrong

Rules are matched by the crawler that reads them, and different crawlers apply Allow and Disallow precedence slightly differently, so a file that behaves as expected for Googlebot can still block a different bot without anyone noticing. Wildcard characters (* and $) are another common source of surprise, because a rule meant to catch one folder can quietly match a pattern across the whole site.

If you need to build a file rather than test an existing one, the Robots.txt Generator covers the common patterns without you having to write the syntax by hand. For a fuller walkthrough of how the directives interact, see the Understanding Robots.txt Tester guide. If the underlying question is really about which pages are being crawled at all, that's a job for a technical SEO crawler.