A friend called me in panic because his online store vanished from Google overnight. The cause was a single line in robots.txt added by a developer during maintenance: Disallow: /. Every page was blocked. A robots.txt tester shows problems like this immediately, before they cost you weeks of traffic.
What does a robots.txt tester do
It reads a website's live robots.txt file and checks whether a specific URL is allowed or blocked for a specific crawler. Robots rules can be tricky because of wildcards, specific user agents and allow rules that override disallow rules. A tester applies the same matching logic search engines use.
How our tester works
The Robots.txt Tester fetches robots.txt from the domain of your URL, finds the rule group for the crawler you selected and applies the longest matching rule, just like Google does. It shows:
- Allowed or blocked for the chosen crawler
- The exact matching rule and the user agent group it came from
- The full robots.txt file for reference
You can test Googlebot, Bingbot, GPTBot or any crawler.
How to use it
- Open the Robots.txt Tester.
- Enter the full URL you want to test, for example https://yoursite.com/blog/post.
- Choose the crawler.
- Click Test URL.
How robots rules are matched
- Specific user agent groups win. If there is a Googlebot group, Googlebot follows it instead of the * group.
- The longest matching rule wins. Allow: /blog/ beats Disallow: / for blog URLs.
- Wildcards like * match any characters and $ marks the end of a URL.
- Empty disallow means everything is allowed.
What to test regularly
- Your homepage
- A main category or service page
- A recent blog post
- A product page
- Any page that suddenly disappeared from Google
If any of these are blocked unexpectedly, fix robots.txt right away. Build a clean file with the Robots.txt Generator.
Blocked does not mean not indexed
A blocked URL can still appear in search results without a description if other websites link to it. If you want a page completely out of Google, allow crawling and add noindex. Confirm everything with the Indexability Checker.
After every site update
Developers often change robots.txt during redesigns, migrations and maintenance. Make it a rule to test your key URLs after every update. It takes thirty seconds and has saved many websites from disappearing.
A real example
An education portal blocked a folder called /course to hide old drafts. Unfortunately the rule also matched /courses/, where all live course pages lived, because robots rules match URL prefixes. Course pages slowly dropped from search. The tester showed the exact matching rule, Disallow: /course, for every course URL. Changing it to Disallow: /course/ fixed the problem. Small details like a trailing slash matter in robots.txt.
Test AI crawler rules too
If you have blocked AI bots like GPTBot in robots.txt, select that crawler in the tester and test a few URLs. Make sure they show as blocked, while Googlebot still shows as allowed. This confirms you protected your content from AI training without accidentally hiding your site from normal search.
Keep a copy of your file
Save a copy of your working robots.txt in a safe place. If a plugin or developer overwrites it by mistake, you can restore the correct version in seconds instead of rebuilding it from memory.
Frequently asked questions
Does Google follow crawl delay
No. Googlebot ignores crawl delay, although some other bots respect it.
What if the site has no robots.txt
Then everything is allowed by default.
Can I test rules before uploading
Upload the file to a test location first, or carefully review the rules generated by our generator.
Test your URLs now with the free Robots.txt Tester.