Robots.txt file is a key element of search engine optimization. It manages the indexing of your website thoroughly. Many of the beginners afraid to code the robots.txt file for their website as incorrect coding may cause the website or blog disappear to search engines.
In fact, the warning message on blogger settings saying “Warning! Use with caution. Incorrect use of these features can result in your blog being ignored by search engines.” would definitely discourage a novice leave his blog’s indexing to air.
Google has introduced a wonderful tool in webmaster tools on Wednesday called as robots.txt file checker that checks out for errors in your robots.txt file and suggests you modifications that are needed.
“Making and maintaining correct robots.txt files can sometimes be difficult. While most sites have it easy (tip: they often don't even need a robots.txt file!), finding the directives within a large robots.txt file that are or were blocking individual URLs can be quite tricky. To make that easier, we're now announcing an updated robots.txt testing tool in Webmaster Tools.” says Asaph Arnon, Webmaster Tools team in Webmaster Central Blog.
Testing robots.txt file
To test your robots.txt file, log in to your webmaster tools and go to your site’s dashboard.
Under Crawl options, select robots.txt tester. You would find the latest version of the robots.txt file of your site like as shown below,
Observe for any warnings or errors on the page and make sure they count zero.
Now, choose your robot from the list and click ‘Test’ to see if it can reach your site or not.
Green is good! If the concerned robot is allowed to access your site, the respective piece of code is highlighted with green along with the turning of ‘Test’ button to ‘Allowed’ status.
Red is not bad! If you don’t want particular bot access your site, you may block it. In this case, you would see a red highlighted piece of code with a ‘Blocked’ turned ‘Test’ button.
Learn how to manage search engine robots in indexing your website.
You can test the robot's code against each and every Google bot that is available from the drop-down list.
Watch a short demo video of the robots.txt file tester tool:
You may be a beginner or a pro, this robots.txt file could help you go easy. If you like the tool, thank Google!
Thank you, Google!
Post a Comment