Robots.txt

[GoPantheoN]

The following Content Is in my robots.txt file .... Is this Correct ?

User-agent: *
Allow: /

 

[DXS]

The correct one is:

User-agent: *
Disallow:

People usually get it confused with:

User-agent: *
Disallow: /

Which disallows robots from looking at all files. Without the "/". It allows the robots to look at all files. (useful for google and such)

 

[Voltar]

If you want to create an in-depth robots.txt and have certain areas included, this might be worth reading.

 

[GoPantheoN]

Thnx guys corrected it

 

[GoPantheoN]

hey plz tell what permission should i set for robots.txt i mean what chmod should i set ?

 

[DXS]

With the CHMOD, I never really bothered with it since my server set it up on it's own to what it should be but I have a feeling 0644 would be good.

 

[GoPantheoN]

ty all finally now robots again started to crawl my site