Robots.txt is a tiny text file with a huge responsibility. One wrong line can hide your entire website from Google. One missing line can let search engines crawl thousands of useless filter pages. I have seen both mistakes on live business websites. Getting this file right takes a few minutes and protects your site for years.
What is robots.txt
Robots.txt is a plain text file at the root of your domain, for example yoursite.com/robots.txt. It tells search engine crawlers which parts of the site they may or may not crawl. It also tells them where your sitemap is. Most good bots respect it.
How our generator works
The Robots.txt Generator builds the file from a simple form:
- Apply rules to all crawlers, or only Googlebot or Bingbot
- Disallow paths you want to block, such as /admin/ or /cart/
- Allow paths to open specific files inside blocked folders
- Crawl delay for bots that support it
- Sitemap URL
- Block AI crawlers such as GPTBot, CCBot, Google-Extended, ClaudeBot and others with one checkbox
The file updates live and you can copy it or download robots.txt.
How to use it
- Open the Robots.txt Generator.
- Add the paths you want to block, one per line.
- Add your sitemap URL.
- Tick the AI crawler option if you want to stop AI training bots.
- Download robots.txt and upload it to your website root.
Common rules for most websites
- WordPress: disallow /wp-admin/ and allow /wp-admin/admin-ajax.php
- Online stores: disallow /cart/, /checkout/ and /my-account/
- Internal search: disallow your internal search path, such as /search, so endless search result pages are not crawled
- Staging sites: block everything, but never forget to remove that rule when you launch
What robots.txt cannot do
Robots.txt controls crawling, not indexing. A blocked page can still appear in Google if other sites link to it. To keep a page out of search results, allow crawling and use a noindex meta tag instead. Check your setup with the Indexability Checker.
Test before you publish
After uploading, test important URLs with the Robots.txt Tester. Make sure your homepage, products and blog posts are allowed and only the intended areas are blocked.
About blocking AI crawlers
Some website owners prefer not to let AI companies use their content for training. Blocking those bots in robots.txt is a simple signal. It does not affect your normal Google search rankings when you block only AI specific bots like Google-Extended rather than Googlebot itself.
A real example
A news website noticed its server slowing down every night. The logs showed bots crawling thousands of tag, search and date archive pages. We added disallow rules for internal search and date archives, kept tags crawlable only for the main topics and listed the sitemap. Server load dropped and Google started spending its crawl time on fresh articles instead of endless archive pages.
Keep it simple
Most small websites need only a few lines in robots.txt: one user agent group, a couple of disallow rules for admin or cart areas and the sitemap line. Complicated files with dozens of rules are harder to maintain and easier to break. If you are unsure whether to block something, it is usually safer to leave it open and use noindex on pages you do not want in search.
Frequently asked questions
Where do I put robots.txt
In the root folder, so it opens at yoursite.com/robots.txt.
Do I need a robots.txt file
It is not required, but it is strongly recommended to point to your sitemap and block useless areas.
Can robots.txt protect private pages
No. It is public and only a request to good bots. Protect private content with passwords.
Create your file now with the free Robots.txt Generator.