Latest Announcemnent
New thread

SharpLeech 1.0.0 (AKA Final)

Status
Not open for further replies.
No idea about the next release. About the suggestion, it already autosaves the last used settings when the app exits. But if the program crashes it doesn't save it (obviously :p).
 
Code: 
   at Hyperz.SharpLeech.Leecher.Leech.GetVbulletinPostInfo(String url, CookieCollection cookies)
   at Hyperz.SharpLeech.MainForm.LeechThreadWorker(Object state)
   at System.Threading.ThreadHelper.ThreadStart_Context(Object state)
   at System.Threading.ExecutionContext.Run(ExecutionContext executionContext, ContextCallback callback, Object state)
   at System.Threading.ThreadHelper.ThreadStart(Object obj)
I'm getting that now :(
 
Alright, heres how it went.

Tried the SEO login, and SharpLeech didn't like it. I just got "login failed" each time. So, I tried the classic login URL and logged in with that - worked fine (as it should anyways).

For posting a new thread, I tried the SEO link for new threads, and when I tried to begin leeching - this is what I got:

m94kdy.jpg
 
Yeah, its definitely the SEO. I'm using it with another forum that doesn't have SEO or CAPTCHA login, and its working perfectly fine.

Waiting on 1.0.2 :D

Awesome work Hyperz, you're the man.
 
In today's news:
IPB3 support has been added, tested and found working. Also added vB 4 support but that's just a layer on top of the vB 3 functionality so nothing special or really new. Just to avoid confusion basically. I also did a test today running 4 instances of #Leech at the same time to get an idea of how fast an asynchronous implementation would be when it would do 4 async calls per loop. Right now SharpLeech does only 1 per loop and that 1 is not asynchronous. The results were pretty much what I expected. I did a similar test in the past so I kinda knew what to expect. Nevertheless, it was fast. Did I mention it was FAST?? Have a look :p:
http://www.youtube.com/watch?v=YLrR2GLi7Pc&fmt=22

Leechers using webbrowser ain't gonna be able to hit this kind of speed, ever. Keep in mind that I only get an effective speed of 480 kbyte/sec out of my connection. Imagine doing 8 async calls per loop... Or 16, 24, lol. You could almost leech the whole of WBB over night. It should go without saying that this isn't very healthy for the site being leeched from and to (high load => DoS effect) >_>.

The video also proves IPB's performance in a way. Since the board in the vid was running on my PC and the CPU usage is visible you can see that bombarding it with new topics has no effect on the performance.
 
will be happy to show you some of the code im using for multiple calls it will make ur life any easier, though i did find that i had to pretty much rewrite my MP app when i went total multi-thread to be sure it was all working effectively.

An easier solution for the use of multi-threading the procedure would be to leech from more than one forum at once, rather than leeching from the same forum, as doing so will trigger some sort of anti-DDOS mechanism, and most likely refuse the connection, as i found myself.
 
Thanks for the offer Jay, but implementing the async calls isn't much of a worry as all I have to do is replace my GetResponse calls with BeginGetResponse and fill in the details. Furthermore I don't want more then one working thread. From what I understand you solved it via additional 'workers' (as in threads). One should be enough, more threads causes unwanted overhead on the parent process :).

See it as this, model 1:
- Process
+- Main thread
+- Another thread
+- Yet another thread
+- More....

I think this is what you used for MP or something along those lines? Would explain why you count the cores.

Model 2
- Process
+- Main thread
+- Working thread => [ non-blocking call 1, non-blocking call 2, ... ]

Non-blocking being the asynchronous call with a callback method. Less overhead and less time spend on syncing a bunch of threads that achieve a similar effect. This would be the way I plan on going about it. Most games work in a similar way where they have one main loop (thread) that handles and shedules all the work. They don't create a new thread if AI #1 walks to point A and another one if an object moves at the same time :p.

In any case, threading is interesting no matter how you go about it. I'm definitely open for 2nd opinions on the matter. Hell, there most likely more models to do the same thing.
 
Hyperz said:
Trying wont hurt.

@captcha: Not at the moment. Some future version might included a webbrowser that can be used if all else fails to grab the cookies and pass them to the 'engine'.
Click to expand...

Even though I already said thanks; double thanks and major kudos to you if you can implement this in a future release.

Thanks again Hyperz, looking forward to the next one. :)

**Edit:
I just saw the new posts in this thread, and I can add to the multiple instances testing. Right now I'm running 6 SharpLeeches at once, everything is running fine.
 
Well i went about it like this:

- Gather Required data (login names, selectedArea, etc) into arrays

And then:

Code: 
for (int i = 0; i < myArray.Length; i++)
    ThreadPool.QueueUserWorkItem(new WaitCallback(BeginSending), (object)i);
Which would trigger the "BeginSending" void. That void would contain somthing like:

Code: 
private void BeginSending(object t)
{
    int ForumCount = (int)t;
    
    WebReq[SendCount].Method = "POST";
    WebReq[SendCount].ContentType = "application/x-www-form-urlencoded";
    WebReq[SendCount].ContentLength = buffer[SendCount].Length;

    .....
    ..... (etc)
}

What that means is i only use one void for every forum type, which makes life a lot easier, but because of the sheer amount of threads flying out, you obviously have to concern yourself with memory/CPU consumption, but onc eyou have figured it all out, you end up with a nice, beautiful little engine. It also means any changes to it are global.
 
That's a 3rd method Jay. Hadn't thought of that of that. I still prefer method 2 though mainly because all calls go trough HttpGET or HttpPOST which are methods that provide some abstraction for the WebRequests. So by rewriting these 2 the whole app can make use off async calls with minimal effort.
 
Status
Not open for further replies.
Back
Top