Reliable, affordable business hosting from Yahoo! Web Hosting
Home Sitemap

SEO ARTWORKS - UNIVERSITY

An ongoing SEO Artworks SEO UNIVERSITY - ALL you need to know about Search Engine Optimization in one Place

META ROBOTS TAG / ROBOT TEXT FILE

Meta Robot Tags are not that important in pages optimization per se but can play a vital role in controlling robots behaviour on your pages. BEST way to tell robots what to do is still your robots.txt file, not your Meta Robot Tage

A robot vists a Web site URL, say http://www.anysite.com but before it does so, it firsts checks for http://www.anysite.com/robots.txt and goes from there.

SEO University Topics:

Please, come back often - this is an ongoing process; we are constantly updating our pages.


Great Analytics Tool:

AffiliateReporting.com 125x125 Banner

 

 

Should you use ROBOTS TXT File or META ROBOTS TAG?

The answer is: you should use BOTH! It is a good practice to include robot.txt file in the root of your website. SEO Artworks site's robots.txt file looks like this:

Sitemap: http://www.seo-artworks.com/sitemap.xml

User-agent: *
Disallow:
Disallow: /contact/

Disallow: /IMAGES/


IMPORTANT: a line below the Sitemap (use Sitemap Generator, the best on-line Sitemap tool to create a SE friendly Sitempapt) reference MUST be empty or the robots.txt file would mess with your pages.

GOOGLEBOT in action:

Googlebot, Google's Robots

In our example above we told to the robot to first go to our Sitemap where it can see all our pages on our site.

User-agent: * means this robots.txt file concerns ALL robots.

Disallow: (empty space) means that all the robots can index all the pages BUT beneath it we forbid robots to visit / index two of our pages: contact and IMAGES pages.

 

You can also forbit a particular robot for visiting your pages:

User-agent: BadRobot
Disallow:

 

ROBOTS META TAG

Robots Meta Tag goes into the head section of your Code:
<head>
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">
</head>

Above example, obviously, forbids robots to index and follow your pages. Basically you have following options (valid values for the "CONTENT" atribute) when creating your Robots Meta Tag:

<META NAME="ROBOTS" CONTENT="NOINDEX, FOLLOW">
<META NAME="ROBOTS" CONTENT="INDEX, NOFOLLOW">
<META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW">


Beware - the robots, especially the bad ones - can search your pages despite your Tags. Also, do NOT use Robots Tag / Robots Txt in order to hide your content from humans. These are easily accessible and intended only for SE.

IMPORTANT: Robots.txt file, correctly located in the root of your site should do the job for you. However you can also use Meta Tag to specify some other issues like precluding robots to use Description Tag from the Open Directory but rather from your own Description Tag. This is how you tell robot NOT to use Open Directory's description for your website snnipet:
<meta name="robots" content="NOODP">

 

The best source for all robots related issues could be find on The Web Robots Page. Before creating your robots.txt or your Meta Robots Tag you should visit it and learn more about specifics.

 

Valid CSS!