Latest Announcemnent
New thread

Help blocking FilesTube/General-Files.com from crawling.

Status
Not open for further replies.
T

tempaa

Active Member
66
2008
6
1,420
Hi guys,

Not sure if anyone here will have a similar issue as I assume most site owners would prefer to have the public download their links.

However my entire site, and all its content is behind a login system. Sites like general-files have somehow recently been able to put my links on their site, for the public to see.

Is there anyone out there who knows the name or IP of their spider/agent so I can block access to sites like these crawling my database for content?


Kind Regards and appreciation in advance.
tempaa
 
8 comments
tempaa said:
Hi guys,

Not sure if anyone here will have a similar issue as I assume most site owners would prefer to have the public download their links.

However my entire site, and all its content is behind a login system. Sites like general-files have somehow recently been able to put my links on their site, for the public to see.

Is there anyone out there who knows the name or IP of their spider/agent so I can block access to sites like these crawling my database for content?


Kind Regards and appreciation in advance.
tempaa
Click to expand...

if i remember right (and your site really is safe)
the crawler can't go behind logged content.
BUT
filestube and general-files should have a personal-crawler, people can install on their computer.
so every site they visit, will be indexed by the PC
this you can't really block

edit:
filestube crawler for PC: http://174.36.205.194/
 
Last edited:
BAD filestube. Oh no they di-nt.
Oh well, maybe one of my users has it installed, and general files has a similar tool.

Ill ask around and see if I can find a way to block things like this.
 
I have a script which blocks almost all unwanted bots/crawlers, and has some pretty good algorithm to detect new ones.

Send me a PM, and we can discuss more on the issue.

tempaa said:
Hi guys,

Not sure if anyone here will have a similar issue as I assume most site owners would prefer to have the public download their links.

However my entire site, and all its content is behind a login system. Sites like general-files have somehow recently been able to put my links on their site, for the public to see.

Is there anyone out there who knows the name or IP of their spider/agent so I can block access to sites like these crawling my database for content?


Kind Regards and appreciation in advance.
tempaa
Click to expand...
 
thanks for the suggestion, but I feel as though there will be a free way to overcome this. I dont really pay for services other than hosting at the moment. If I am unable to do this myself, I will be in touch.

Kind Regards
 
Status
Not open for further replies.
Back
Top