In General, The Robots.txt is a file that sits at the root level of our website and helps to block search engine bots to index and crawl important part of our site or blog. Hence it’s must to make full use of it by going through correct settings for good SEO. Usually people don’t know about Robots.txt file and how to control it. Well never try to play with your Robots.txt file without any proper knowledge because anything wrong will let your blog go away from search engines. So today I am going to provide you a readymade SEO optimized/Friendly Robots.txt file that suites for all themes below. Have a look!
Settings for an Ideal SEO optimized WordPress Robots.txt file:
Sitemap: https://www.bloggingways.net/sitemap.xml User-agent: * Disallow: /cgi-bin/ Disallow: /wp-admin/ Disallow: /wp-includes/ Disallow: /wp-content/ Disallow: /archives/ Disallow: /*?* Disallow: /category/ Disallow: /tag/ Disallow: /search Disallow: /index.php Disallow: /xmlrpc.php Disallow: *?replytocom Disallow: /wp-* Disallow: /author Disallow: /feed/ Disallow: /trackback/ Disallow: /comments/feed/ Disallow: /page/ Disallow: /go/ Disallow: /comments/ Disallow: /wp-content/plugins User-agent: Mediapartners-Google* Allow: / User-agent: Googlebot-Image Allow: /wp-content/uploads/ User-agent: Adsbot-Google Allow: / User-agent: Googlebot-Mobile Allow: /
Important Note– Don’t Forgot to replace www.bloggingways.net with your Blog url above in the sitemap.
Benefits of this Robots.txt File Settings:
1. Protects your blog from Duplicate Content issue.
2. Noindex your WordPress Categories and Tags from Search engines.
3. Protect your Admin files to get listed in search results.
4. Your WordPress plugins get hidden from search engine results.
5. Other things such as Comments, feed, trackbacks gets excluded from Google spiders.
6. Providing your WordPress media i.e Images clear way to get listed in Google Images fastly.
7. Saves your blog from Google Panda Effect.
8. Make your blog more SEO friendly etc.
How to Find and Edit your WordPress Robots.txt file?
You can find your Robots.txt file inside your root level Directory of Cpanel. Now to Edit your WordPress Robots.txt file, You can either edit it by logging into your FTP account of server or you can use plugin like KB Robots to edit robots.txt file from your WordPress dashboard as well. That’s it!
How to Check your Robots.txt made Correct?
Once you have made all the changes to your WordPress Robots.txt file, its time to check it via Google Webmaster Tools. Just login to that and click on your site and then choose Health>> Fetch as Google. Now simple enter your any post url to check if there is any issue accessing your post in Google.
Extras- 1. You can also check for the crawl error caused due to Robots.txt file under Crawl error section of Health menu.
2. You can also check for Blocked url caused due to Robots.txt file under Blocked url’s section of Health menu.
I Hope this Robots.txt file provides you the best settings for your blog considering SEO in mind. Enjoy and please comment below if you have any doubt or queries related to this post. Best of Luck.
How to Make your WordPress Robots.txt File SEO Optimized
You can visit our website to get https://imrantech.site/affordable-local-seo-services/