rickymartin
Active Member
Is there any other option to prevent crawler from crawling a site or page except robot.txt.
[COLOR=#881280][FONT=monospace]<meta [/FONT][/COLOR][FONT=monospace]name[/FONT][COLOR=#881280][FONT=monospace]="[/FONT][/COLOR][FONT=monospace]robots[/FONT][COLOR=#881280][FONT=monospace]" [/FONT][/COLOR][FONT=monospace]content[/FONT][COLOR=#881280][FONT=monospace]="[/FONT][/COLOR][FONT=monospace]noindex, nofollow[/FONT][COLOR=#881280][FONT=monospace]" />
[/FONT][/COLOR]