Which one is better Robots.txt or Meta robots Tag ?

[vgkumar]

Which one is better Robots.txt or Meta robots Tag ?

 

[semaphore.v]

Robots.txt

 

[arindamroy]

Hello,
I think robots.txt is better than meta robots tag. If you use meta robots tag then you have to use this tag in each and every web page.
On the other hand, only one robots.txt file needed for entire website.

 

[AmN]

Both are supported by all crawlers which respect webmasters wishes. Not all do, but against them neither technique is sufficient.

You can use robots.txt rules for general things, like disallow whole sections of your site. If you say Disallow: /family then all links starting with /family are not indexed by a crawler.

Meta tag can be used to disallow a single page. Pages disallowed by meta tags do not affect sub pages in the page hierarchy. If you have meta disallow tag on /work, it does not prevent a crawler from accessing /work/my-publications if there is a link to it on an allowed page.

 

[rickymartin]

Robot.txt is the better Because you can write all code in a single file while if you use robo meta tag then you will have to use it on each page where you want to use it.

 

[nepstermartin]

Both are similar and both provide same facility so you can use any of it as per your requirement. When you have so many pages to not index, robots.txt file is more continent.