What a robots.txt file actually controls
A robots.txt file is a plain text document sitting at the root of your domain, at yourdomain.co.uk/robots.txt, that tells well-behaved crawlers which paths they're welcome to request and which to leave alone. It doesn't stop a page appearing in Google's index if that page is linked from somewhere else, and it doesn't secure anything: any browser can open the file and read it. What it does is manage traffic from crawlers, which matters most on larger sites where you'd rather Googlebot spent its time on product pages than on filtered search results or admin routes.
The Robots.txt Generator builds that file for you from a set of rules you choose: which user-agent each rule applies to, which paths are disallowed or explicitly allowed, and where your sitemap lives. The value isn't the typing it saves. It's that the syntax has sharp edges, a wildcard in the wrong place or a missing trailing slash changes what gets blocked, and a generator applies the pattern consistently.
Crawl budget and why the rules matter
Crawl budget (the number of pages a search engine is willing to fetch from your site in a given period) is finite, and it's spent unevenly if you let it be. A large ecommerce site with faceted navigation can generate thousands of near-duplicate URLs from filter combinations alone, and every one of those a crawler fetches is a page it didn't spend fetching something you actually want ranked. Disallowing those paths doesn't guarantee they're excluded from search, but it does stop crawlers repeatedly requesting them, which is the more common reason sites use robots.txt in practice.
This is really only worth doing deliberately on sites with scale: thousands of URLs, faceted or parameterised navigation, or staging environments that shouldn't be crawled at all. A five-page brochure site gains little from elaborate rules and more from making sure the file doesn't accidentally block the whole thing.
Mistakes worth checking for
- Blocking CSS and JavaScript by accident: a broad disallow rule aimed at an admin folder can catch shared asset paths too, which stops Google rendering the page as a visitor sees it.
- Treating disallow as noindex: disallowing a path stops it being crawled, but if the URL is linked from elsewhere it can still appear in search results without a description, since Google never fetched the page to know what's on it.
- Wildcard patterns that catch more than intended: an asterisk placed one character too early or late can disallow a whole directory.
- Case sensitivity: paths in robots.txt are case-sensitive, so /Admin/ and /admin/ are different rules as far as a crawler is concerned.
- No sitemap directive: including the sitemap location in robots.txt is a small addition that gives crawlers a direct route to your full URL list.
Checking the file before it goes live
A file the generator produces can be syntactically correct and still block the wrong thing, because the tool follows the rules you give it. It's worth running the finished file through the Robots.txt Tester against a handful of real URLs from your site before you publish it, checking that the pages you want crawled are marked as allowed and the ones you don't are marked as blocked. Like any automated check, the tester can return a result that doesn't match what actually happens once a crawler hits your live site, so treat it as a strong indication rather than a guarantee.
When to build the rules by hand instead
The generator covers the common cases well: disallowing a handful of paths, allowing exceptions within a disallowed folder, and pointing to a sitemap. Where it gets more involved is sites with several subdomains each needing different rules, or where you want separate treatment for AI crawlers alongside traditional search bots, a distinction covered in more detail on our AI search visibility tools comparison. For sites large enough that crawl behaviour needs ongoing attention rather than a one-off file, pairing robots.txt with a proper crawl of your own site, using one of the tools on our technical SEO crawlers comparison, gives you a clearer picture of what's actually being requested rather than what you assume is happening.