Crawlers

[daveborg]

Anyone know any good crawler scripts? Looking for something I can mod to look for my own criteria.

 

[glcon]

ask mRAza

 

[daveborg]

Thanks I will, looking for something really flexible not sure it exists

 

[BaqarNaqvi]

I am still unaware of the crawler purpose 8-)

 

[Daniel]

^ http://en.wikipedia.org/wiki/Web_crawler

 

[l0ck]

the best crawler is google crawler :p
just kidding

 

[BaqarNaqvi]

^ http://en.wikipedia.org/wiki/Web_crawler

Thanks

 

[daveborg]

No worries I've built one.

 

[risi]

I remember, I used indexu before and it was satisfying my needs. However web crawlers are very heavy applications as they store to much in database and it is always growing having it on shared hosting won't get you very far. after some time you will need more specific configuration on your server.

 

[daveborg]

Sure, Indexu is really what I'm looking for, I need something more complex so I can target the info I need and have it brought back, guess there's nothing out there, so time to crack open the php.