Robots.txt Tester
Check whether Google and other crawlers may fetch a page of your domain, and see whether your sitemap can be read without errors.
Result for your address
Your robots.txt
Show the whole file
| Line | Content |
|---|
Your sitemaps
How to go about it
Enter the address
Enter the page you want to check. The tool fetches the robots.txt by itself.
Choose a crawler
Choose the crawler the rules should apply to. Each crawler only reads its own section.
Read the result
You see whether the page may be fetched, which line decides it and how your sitemaps are doing.
What the robots.txt is
The robots.txt is a plain text file in the root directory of your domain. In it you decide which parts of your website search engines and other crawlers may fetch. It always sits at the same address: your-website.com/robots.txt.
Every section starts with a User-agent line that says who it applies to. The rules follow below it: Disallow blocks a path, Allow opens it up again. The asterisk stands for any characters, a dollar sign at the end for the end of the address.
How Google reads the rules
A crawler only reads the section that carries its name. If there is none, the section with the asterisk applies. So Googlebot no longer reads the rules under User-agent: * as soon as there is a section of its own for Googlebot.
Within the section it is not the order that decides but the length: the rule with the longest matching path wins. If two rules are equally long, Allow wins. That is exactly why the tool tells you the line that decided.
If there is no robots.txt, everything may be fetched. If the server answers with a 5xx error, Google treats the whole domain as blocked for the time being.
What makes a good sitemap
A sitemap is an XML file that lists all the pages that should be found. Add it to your robots.txt with a Sitemap line, and every search engine finds it without any further submission.
A sitemap holds up to 50,000 addresses. It contains only full addresses of your own domain, and the robots.txt allows each of them to be fetched. If a page is in the sitemap and blocked at the same time, the two files contradict each other.
Larger websites split their addresses across several sitemaps and combine them in a sitemap index. The tool lists the sitemaps of an index, and you can check each one individually.
