So i got two brand new domains with zero backlinks. Each has the exact same number of links per page with a similar layout and targeted Keywords. The script is custom but based on a similar layout to Wordpress. I gave each domain the same amount of limited in links so Google could find the site.
So basically two small pretty much identical sites with the same amount of content.
The only difference is one has a sitemap and rss feeds that Google can use and the other doesn't have any. For a few days it was that I forgot to enable it but then I though I'd leave it for a while longer to see the difference.
The difference is massive. Google crawls the site that has a sitemap and rss feeds 5 times more than the one without a sitemap and growing at a much faster rate.
Now everyone tells you to add a site map because it's better for your site but I never expected it to have this much of an impact.
So what do you do now? Well it you don't have a sitemap your killing yourself. Get one now!
Now make a robots.txt file and upload it to the root of your site.
I'm not really interested in what you allow or disallow Google to crawl you can just Google that stuff yourself but at the end of the sitemap I want you to add a link to the sitemap as it helps search engines find it way way faster.
If you have RSS feeds (which you should and if you don't get them) then you can add them too like below. Change the links to what ever your RSS feeds are.
Seriously this will really help your site. I'm amazed how much of a difference it makes. Once you've it add in the robots.txt file you can see that it will be accepted in the results in the Google Webmaster Area. They will find it in less than a day usually and you don't even have to manually submit it.
[slide]http://screensnapr.com/u/u46vbu.png[/slide]
If you don't have a Google Webmaster Area your missing out too. Get one of them straight after reading this topic as the info they have is invaluable.
Help me spread the message. When your reviewing a site. Don't just say get a favicon. Tell them to add a sitemap too if they don't have one.
Anyone who asks me for help in future won't get any unless they have a sitemap :P
Any questions let me know
EDIT:
Another tip. If you have a huge site you can brake up your sitemaps. Google has limits on how big a sitemap should be which is no more than 50000 urls and a max 10MB file size (after un-gzipping).
Here's an example of a site and the way I split them up.
So basically two small pretty much identical sites with the same amount of content.
The only difference is one has a sitemap and rss feeds that Google can use and the other doesn't have any. For a few days it was that I forgot to enable it but then I though I'd leave it for a while longer to see the difference.
The difference is massive. Google crawls the site that has a sitemap and rss feeds 5 times more than the one without a sitemap and growing at a much faster rate.
Now everyone tells you to add a site map because it's better for your site but I never expected it to have this much of an impact.
So what do you do now? Well it you don't have a sitemap your killing yourself. Get one now!
Now make a robots.txt file and upload it to the root of your site.
I'm not really interested in what you allow or disallow Google to crawl you can just Google that stuff yourself but at the end of the sitemap I want you to add a link to the sitemap as it helps search engines find it way way faster.
Code:
User-agent: *
Sitemap: http://example.com/sitemap.xmlIf you have RSS feeds (which you should and if you don't get them) then you can add them too like below. Change the links to what ever your RSS feeds are.
Code:
User-agent: *
Sitemap: http://example.com/rss/
Sitemap: http://example.com/rss/app/
Sitemap: http://example.com/rss/movie/
Sitemap: http://example.com/rss/game/
Sitemap: http://example.com/rss/music/
Sitemap: http://example.com/rss/tv/
Sitemap: http://example.com/rss/ebook/
Sitemap: http://example.com/sitemap.xmlSeriously this will really help your site. I'm amazed how much of a difference it makes. Once you've it add in the robots.txt file you can see that it will be accepted in the results in the Google Webmaster Area. They will find it in less than a day usually and you don't even have to manually submit it.
[slide]http://screensnapr.com/u/u46vbu.png[/slide]
If you don't have a Google Webmaster Area your missing out too. Get one of them straight after reading this topic as the info they have is invaluable.
Help me spread the message. When your reviewing a site. Don't just say get a favicon. Tell them to add a sitemap too if they don't have one.
Anyone who asks me for help in future won't get any unless they have a sitemap :P
Any questions let me know
EDIT:
Another tip. If you have a huge site you can brake up your sitemaps. Google has limits on how big a sitemap should be which is no more than 50000 urls and a max 10MB file size (after un-gzipping).
Here's an example of a site and the way I split them up.
