What is Robot.txt?

Robot.txt is like one of the protocol that control the communication between web and server. It is used to access all sites or to access any particular site or deny all sites and we can set delay process also. For Example. If you are owner for the site http://www.mysiteranksfirstingoogle.com/ means you want to hide some inner pages in your site or you want to set other users should not follow your site that is “no follow”, default  web spider will visit all site inner pages. To control this we can set the control in robot.txt file. If you want to protect your site means www.mysiteranksfirstingoogle.com/robot.txt. Do some alteration in the notepad file like

There are three different user agents names are there for search engine

Google – Google bot

Yahoo – slurp

Bing – msnbot

Webspider will allow to crawl all sites

Useragent:*

Disallow:

Allow:/

For to disallow

Useragent:*

Disallow:/

For to disallow only login page in google means

Useragent:Google obt

Diallow:/login.htm/

I think this information will be useful for those who are searching for robot.txt

This will bring my site ranks first in google.