How to Edit your robots.txt file in new Blogger

If you have been on blogger for a while you will know that you couldn't directly edit your robots.txt file. The only way to change it was to use robots meta in your template. Now with the new upgrade to blogger we have access to this little but very important file. We can now change our robot.txt file from within our dashboard.

For people that don't already know what a robots.txt is , here's a little explanation of what they are and how they are used within your site.

What is a robots.txt file?

The Robot Exclusion Standard, also known as the Robots Exclusion Protocol or robots.txt protocol, is a convention to prevent cooperating web crawlers and other web robots from accessing all or part of a website which is otherwise publicly viewable.

Robots are often used by search engines to categorise and archive web sites, or by webmasters to proofread source code. The standard is different from, but can be used in conjunction with, Sitemaps, a robot inclusion standard for websites.

Why use a robot.txt file?

Sometimes you may want to stop certain parts of your site being indexed in search engines e.g. archive pages of a blog. The easiest way to let search engines know that you don't want these pages crawled is to use a robots.txt. You can set different rules for different robots and web crawlers depending on your preferences.

But, not all crawlers will listen to your robots.txt file and continue to access your pages, robots.txt simply asks not to be crawled it doesn't block access. Most big crawlers will follow your requests.

What goes into a robots.txt file?

There are various things you can put into your robots.txt file. Here is an example of a basic robots.txt file...

User-agent: * Disallow:
The example above tells all robots to visit all files because the wild card * specifies all robots.

User-agent: * Disallow: /
The example above tells all robots to stay out of a website.

You can also tell robots to not crawl specific directories. It would look something like this...

User-agent: * Disallow: /cgi-bin/ Disallow: /images/ Disallow: /tmp/ Disallow: /private/

For more information about robots.txt and what can go in it check out the wiki here... http://en.wikipedia.org/wiki/Robots_exclusion_standard

How do you edit your Blogger robots.txt file?

Well, you cant really edit your Blogger robots.txt but you can use your own custom robots.txt info.

Blogger already has a robots.txt file but you cant edit it. When you create your custom robots.txt it will override the default rules and display your custom robots.txt info.

To create a custom robots.txt in new blogger...

1. Log into your dashboard and navigate to Settings and then in the dropdown menu select Search preferences.
2. At the bottom of the page you will see a section called Crawlers and indexing, in that section is Custom robots.txt.
3. Hit the Edit button on the right and enable it.
4. Enter you custom robots.txt rules and save changes. Done

Im not going to tell you exactly what to put in your  custom robots.txt, because each site is different and needs different rules. Before you start using robots.txt make sure you have properly researched the necessary rules for your type of site and you know exactly what you want crawled and not crawled.

Warning

Improper use can prevent your site from being indexed properly in search engines, so make sure you read a bit more about it and know what you are doing before proceeding.