Set Robot txt File For Blogger Blog For More Search Traffic

Did you know you can improve your search traffic (organic traffic) by setting a simple, but the best robot txt file for your blog by increasing the number of posts being indexed and blocking some irrelevant pages to appear on Google search engine?

That is, you can change the number of posts you want Google to index, and also stop some certain pages from being indexed by Google.

Exactly!

Robot.txt file is used to inform robots (spiders, crawlers, etc. sent by search engines) about your blog, its structure and to tell whether or not to crawl your blog posts or pages on your blog.

You as a blogger would like individual posts or pages of your site to be indexed and crawled by search engines, while you may prefer other irrelevant pages not to be crawled.

My robot.txt file may be different from your own, depending on choice.

However, for the sake of being a newbie, I don't want you to lay your hands on editing many codes because incorrect use of this code below may result in your blog being ignored by search engines.

So, I have made it easy for you by giving you a simple file you can easily edit with your blog link ONLY.


Below is the method to set up the best robot.txt file you should use:

==> On your blogger dashboard, go to “Settings” >> “Search preference” >> “Crawlers and Indexing”

==> Click the “Edit” button near the “Custom robot.txt” and check the “Yes” checkbox

==> Paste the code below:

User-agent: Mediapartners-GoogleUser-agent: *
Disallow: /search
Allow: /
Sitemap:http://easymadeblog.com/atom.xml?redirect=false&start-index=1&max-results=500
Sitemap:http://easymadeblog.com/atom.xml?redirect=false&start-index=1&max-results=1000

==> Change the “easymadeblog.com” in red ink to your blog address. Then save

The sitemap I used above will let Google search engines index up to 1000 posts on your blog, unlike the default which is just 25 posts.

• Read: 15 Tools That Make Blogging Easier To Do

But in case you don't want a particular post or page to be indexed,

Simply change the "search" in line 3 of the code to the post or page permalink. It should look like this:

User-agent: Mediapartners-GoogleUser-agent: * 
Disallow: /year/month/post-permalink 
Allow: / 
Sitemap:http://easymadeblog.com/atom.xml?redirect=false&start-index=1&max-results=500 
Sitemap:http://easymadeblog.com/atom.xml?redirect=false&start-index=1&max-results=1000

OR

User-agent: Mediapartners-GoogleUser-agent: *

Disallow: /p/page-url.html

Allow: /

Sitemap:http://easymadeblog.com/atom.xml?redirect=false&start-index=1&max-results=500

Sitemap:http://easymadeblog.com/atom.xml?redirect=false&start-index=1&max-results=1000

This is an example of one of my posts below:

http://www.easymadeblog.com/2017/05/make-money-from-blogging.html

And a page on my blog:

http://www.easymadeblog.com/p/write-for-us.html

Let's assume I don't want either of the above to be indexed by Google search engine, I will just copy the highlighted text above and replace with the "search" as it was explained above.

And this way, you're done!

• Read also: Submit Sitemap To Yandex And Bing Webmaster Tools For More Traffic

Conclusion:

Use the robot txt file with caution in order not to make your blog being ignored by Google search engines.

Let me know if you would like me to help you set the robot txt file for your blogger blog.