Get Started

Robots.txt Generator

Choose which bots to block, list any folders search engines should not crawl, add your sitemap, then generate a ready-to-upload robots.txt file. Once created, check it with our Robots.txt Tester.

Block specific crawlers entirely

Ticked bots will be fully disallowed from your whole site. Leave unticked to allow them, subject to the rules below.

Default rule for all other crawlers
Folders to disallow

Only used when "Allow all other bots" is selected above. One path per line, each starting with a forward slash, for example /wp-admin/

Sitemap and crawl rate

Applied to the "User-agent: *" rule whether it allows or blocks other bots, since crawl-delay is still honoured by some crawlers even on a blocked wildcard rule.

Your robots.txt