Everything about Meta Robots and robots.txt

Robots plays important role in the field of SEO. We have two ways to control pages and folders one is using Robots META tag and other is through robots.txt

A web page creator can specify which page should be indexed and which page should not be indexed by search engines by placing Robots META tag in the HTML section

Here are some Robots tags that are common

< content="NOINDEX" name="ROBOTS">- Ignore content and follow links
< content="NOFOLLOW, INDEX " name="ROBOTS">- Include content and do not follow links
< content="NOINDEX,NOFOLLOW" name="ROBOTS">- Ignore content and do not follow links
< content="INDEX,FOLLOW" name="ROBOTS">- Include content and follow links
< content="NOARCHIVE" name="ROBOTS">- Cache link should not show Search results pages
< content="NOODP" name="ROBOTS">- The Open Directory Project (ODP) title and description for the page should not be displayed in Search results
< content="NOYDIR" name="ROBOTS">- The Yahoo Directory title and description for the page should not be displayed in Search results
< content="NOSNIPPET" name="ROBOTS">- Titles are only displayed in Search results page and not description or text context for this page

In addition to manage folder level user agent control robots.txt file can be used. This file can be placed in root of each server and the format is plain text not HTML

Through this file website owner or webmaster can allow access to web page content and disallow access to admin, cgi and any secured files that you don’t want search engines to index

A typical robots.txt file will look like

User-agent: *
Allow: /
Disallow: /admin*
Allow: *content*
Disallow: /test/
Disallow: /paypal/
Disallow: /credit/
Disallow: /cgi/

Explains, all robots can crawl except the admin files, and crawl files named content folder, and should not crawl test, paypal, credit and cgi folder.

Hope this post helps to know more about robots and its act in SEO. You can analyze further on checking Google's robots.txt file http://www.google.com/robots.txt and post comment or tweet @jagadeeshmp if you need better understanding :)

30 SEO Tips in 30 Days to turn your website SEO ready

Revised 30 SEO Tips in 30 Days - Explained in Google Knol (Won best knol of the Month March 2009 & Top Pick Knol Award)

Having less time to do SEO by your own :) do not worry, now you have a chance to turn your website SEO ready in just 30 days by applying one SEO tip a day? Here are those 30 SEO tips

Tip [1] Add Titles related to targeted keywords, place only important keywords, do not exceed more than 25 characters
Tip [2] Meta description should be unique for each pages with targeted keywords. Do not stuff keywords into Meta description.
Tip [3] Layout pages with CSS, avoid tables
Tip [4] Add Keywords towards H1 tag as page headlines.
Tip [5] Implement Bread crumbs for increasing web site navigation structure example Home >> Service >> SEO
Tip [6] Create Static Header and footer navigation links, Links should be text not as images
Tip [7] Analyze your targeted keywords using Google suggest
Tip [8] Keywords in navigational links play a vital role and have its own importance to search engine eyes
Tip [9] Use most important keyword in file name or URL
Tip [10] Minimize table especially nested tables
Tip [11] Say yes to Quality links and No to Quantity links
Tip [12] Never try to place hundreds of link for a new site on the very first day
Tip [13] Your targeted keywords need to vary a lot in anchors
Tip [14] Always have a good combination of links from new and old sites
Tip [15] Try writing articles having links and have them in the resource page of your site
Tip [16] Images are important, make sure your site logo contains your keyword
Tip [17] Put CSS and JavaScript into external file / folder
Tip [18] Pay special attention to your content
Tip [19] Make sure to have sitemap as text links
Tip [20] Place robots.txt in root on your server
Tip [21] Create Google site map to your web site
Tip [23] Start a blog and participate with other related blogs
Tip [22] Submit site to open directories
Tip [24] Create feed xml and ror xml for xml search engines and directories
Tip [25] Try to avoid automatic directory submission and build manual links
Tip [26] Monitor your hits and traffic by using free Google analytics
Tip [27] Know how to analyze the reports and to optimize the results
Tip [28] Know about redirection methods and theory - 301, 302, and 200 response codes
Tip [29] Implement, understand and react to website analytics reports
Tip [30] Make your body copy sufficiently long and keyword rich

Quick Guide

If you want these SEO tips in your site or blog as Gadget, feel free to use my SEO Tips of the Day Google Gadget that displays one SEO tip a day in your web page which can even help your site visitors to learn SEO tips and to get SEO and other Internet marketing Tweets follow me @jagadeeshmp , thanks for your time!