robot.txt?

[peopletrendin]

what is meant by robot.txt in seo?

 

[OneManCrew]

The robots.txt is a text file you upload to the root of your site which guides and communicates with web crawlers.
With this file you can direct the robot crawler which areas of the website they should not scan or do scan.

Examples:

Allow visit to all files:
User-agent: *
Disallow:

Disallow the following directories:
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /junk/

You can find a great tool to create the robots.txt file on WJunction - WJunction - Webmaster Forum - Wembaster Tools

 

[Upmoney]

what is meant by robot.txt in seo?

Also you can look at the useful Google guide about robots.txt.

 

[TheSHosting]

Just to add, note that rogue bots does not honor robots.txt files. So you will have to use other methods to block them and do not attempt to use robots.txt to block such agents.

 

[tushardave]

Robot.txt File

If you don't want some of urls from your site to craweled by search engines put those pages in robot.txt file