Learn and Share about SEM in Search Engine Watch Forums

Hello world, a short information about search engine watch forums and my part.
I would like to share the information that SEW forums is one of the place where I had spent time to learn more and in-depth about SEM industry years ago. There were several topics, which has been posted in search engine watch forums that was helpful and even now it goes on and on. Latter I had chance to reply threads by sharing my knowledge and it was a great pleasure to know that those thread replies were helpful for most of the members.

There are many Moderators and members who actively participate the forum thread posts and comes up with great advice, tips and techniques which tends to bring out their expertise knowledge in specific filed and also in general. I am very much happy to add value for SEW forums and I am glad to say that I am one of the Moderators

If you are a site owner, webmaster, designer, developer or a SEO person trying to improve your ability in search engine marketing, then the right place is Search engine watch forums. I welcome you to Register and get in touch with SEM queries!

Everything about Meta Robots and robots.txt

Robots plays important role in the field of SEO. We have two ways to control pages and folders one is using Robots META tag and other is through robots.txt

A web page creator can specify which page should be indexed and which page should not be indexed by search engines by placing Robots META tag in the HTML section

Here are some Robots tags that are common

< content="NOINDEX" name="ROBOTS">- Ignore content and follow links
< content="NOFOLLOW, INDEX " name="ROBOTS">- Include content and do not follow links
< content="NOINDEX,NOFOLLOW" name="ROBOTS">- Ignore content and do not follow links
< content="INDEX,FOLLOW" name="ROBOTS">- Include content and follow links
< content="NOARCHIVE" name="ROBOTS">- Cache link should not show Search results pages
< content="NOODP" name="ROBOTS">- The Open Directory Project (ODP) title and description for the page should not be displayed in Search results
< content="NOYDIR" name="ROBOTS">- The Yahoo Directory title and description for the page should not be displayed in Search results
< content="NOSNIPPET" name="ROBOTS">- Titles are only displayed in Search results page and not description or text context for this page

In addition to manage folder level user agent control robots.txt file can be used. This file can be placed in root of each server and the format is plain text not HTML

Through this file website owner or webmaster can allow access to web page content and disallow access to admin, cgi and any secured files that you don’t want search engines to index

A typical robots.txt file will look like

User-agent: *
Allow: /
Disallow: /admin*
Allow: *content*
Disallow: /test/
Disallow: /paypal/
Disallow: /credit/
Disallow: /cgi/

Explains, all robots can crawl except the admin files, and crawl files named content folder, and should not crawl test, paypal, credit and cgi folder.

Hope this post helps to know more about robots and its act in SEO. You can analyze further on checking Google's robots.txt file http://www.google.com/robots.txt and post comment or tweet @jagadeeshmp if you need better understanding :)