Latest Announcemnent
New thread

Robots.txt file code, is it right?

Status
Not open for further replies.
5 comments
In that way bots will scan all files.

What's the url of your website?
What script are you running?

You can use disallow to prevent bots crawling files that don't need to be crawled.

This is how I have the robots.txt, I'm using phpbb.

Code: 
User-agent: *
Disallow: /phpbb/viewtopic.php
Disallow: /phpbb/viewforum.php
Disallow: /phpbb/index.php?
Disallow: /phpbb/posting.php
Disallow: /phpbb/search.php?
Disallow: /phpbb/ucp.php
Disallow: /phpbb/mcp.php
Disallow: /phpbb/post

Hope it helps you.
 
robots.txt

Google robots.txt for vbulletin, don't forget you are using friendly urls.

You can use some like this:

Code: 
User-Agent: *
allow: /
Disallow: /cgi-bin/
Disallow: /ajax.php
Disallow: /attachment.php
Disallow: /calendar.php
Disallow: /cron.php
Disallow: /editpost.php
Disallow: /global.php
Disallow: /image.php
Disallow: /inlinemod.php
Disallow: /joinrequests.php
Disallow: /login.php
Disallow: /member.php
Disallow: /memberlist.php
Disallow: /misc.php
Disallow: /moderator.php
Disallow: /newattachment.php
Disallow: /newreply.php
Disallow: /newthread.php
Disallow: /online.php
Disallow: /poll.php
Disallow: /postings.php
Disallow: /printthread.php
Disallow: /private.php
Disallow: /profile.php
Disallow: /register.php
Disallow: /report.php
Disallow: /reputation.php
Disallow: /search.php
Disallow: /sendmessage.php
Disallow: /showgroups.php
Disallow: /subscription.php
Disallow: /threadrate.php
Disallow: /usercp.php
Disallow: /usernote.php

You can add a sitemap line after the last line Disallow: /usernote.php:

Code: 
Sitemap: http://www.hangout.com/sitemap.xml.gz


---------- Post added 16th Mar 2013 at 12:17 AM ---------- Previous post was 15th Mar 2013 at 11:15 PM ----------
RDB said:
hi,

i am using:
Code: 
User-agent: *
Allow: /
in my robots.txt....what it means and what is its function?
Click to expand...

it means you allow bots to crawl all your files.
 
Last edited:
Nice to see all post just i suggest all again go to Google web master tools and then generate once more time and add it in your website ...
 
Status
Not open for further replies.
Back
Top