You have a new web site. You install the wordpress and start to develop your web page but you don’t want to any search engine start to index your web site and during the wordpress installation or just after from Settings > Reading Pane you select the ” Discourage search engines from indexing this site ” setting.
You finished developing your site and you are ready to go online and it is time to let the search engines to index your site. And you disable the ” Discourage search engines from indexing this site” setting.
And you sign up for Google Web Master Tools to manage your site existence in Google more better. The first thing to do is generate sitemap of your site and add it to to Web Master Tools. Your search for a siyemap generator plugin and you find one the most popular wordpress sitemap generator plugin is Google XML sitemaps. You install the plugin and add your sitemap.xml in usual way. And you get a warning message “URL blocked by robots.txt” warning message in Webmaster tools.
How to solve the URL blocked by robots.txt warning message in Webmaster tools?
You google “URL blocked by robots.txt warning message” and find that this problem is solved. That mention to reset your robots.txt and generate new file in your root directory and put below code into it.
You try to resubmit your sitemao to web master tools. But same warning “URL blocked by robots.txt“.
Reinstalling Google XML sitemaps plugin this will also not the solution.
So what is the solution for fixing the “URL blocked by robots.txt warning message”.
Infact it is very simple. Just go to your Web Master Tools Dashboard.
Crawl > Robots TXT Tester
And then click “See live robots.txt” on the right site of the screen and copy all and return to Web Master Tools screen and erase the last version of the robots.txt and paste the live version and click the submit button.
Resubmit your sitemap and you will see that “URL blocked by robots.txt warning message” has been solved.