[c#] CHALLENGE - Beat the time!

[jayfella]

Ok.
So Hyperz and i were discussing the finer points of multi-threading, asynchronous threads and all that story.

We decided to carry out a friendly "code-off" and see who could come up with the fastest solution based on how you concluded the best way to go about doing such a thing:

The task is simple: Perform a series of (pre-decided) http requests and time the request responses. The fastest code wins. Simple. We are also using a HTTP time-out of 5000 milliseconds, to avoid dead/slow servers from hampering the results.

URL LIST: http://www.coding.extremecoderz.com/urls.txt


Let the battle commence! In a FRIENDLY way! Please DO NOT flame if you get pwnt, this is in the interest of fun and learning.

Please Also Note: We are using a console application, not a windows Forms Application.

 

[jayfella]

MY code:

Im not a man of the book. I just write it how i see it. This probably goes against every rule in the programming book. Still. Its fast as fuck.

Basically what im doing here is throwing async out the window. I chuck everything into one huge array and throw it at a threadpool, which houses an array of HTTP-Repquests. Simplicity ftw...?

using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;
using System.IO;
using System.Text;

using System.Threading;
using System.Net;

namespace ConsoleApplication1
{

    class Program
    {

        public static string[] myUrls;

        // HTTP arrays
        static HttpWebRequest[] webReq;
        static HttpWebResponse[] webResp;
        static ManualResetEvent[] resetEvents;

        static Stopwatch sw = new Stopwatch();

        static void Main(string[] args)
        {
            sw.Start();
            Console.WriteLine("starting...");
            RunTest();
        }

        public static void RunTest()
        {
            string s = "";
            using (StreamReader rdr = File.OpenText("urls.txt"))
            {
                s = rdr.ReadToEnd();
            }

            myUrls = s.Split('\n');

            // Set Min-Max Threads (setting Both the same removes time-to-create delays)
            ThreadPool.SetMinThreads(100, 100);
            ThreadPool.SetMaxThreads(100, 100);

            // Declare array Sizes Before starting...
            DeclareArrays();

            for (int i = 0; i < myUrls.Length; i++)
            {
                resetEvents[i] = new ManualResetEvent(false);
                ThreadPool.QueueUserWorkItem(new WaitCallback(DoRequest), (object)i);
            }
            
            Console.Read();
        }

        public static void DeclareArrays()
        {
            webReq = new HttpWebRequest[myUrls.Length];
            webResp = new HttpWebResponse[myUrls.Length];
            resetEvents = new ManualResetEvent[myUrls.Length];
        }

        public static void DoRequest(object o)
        {
            int i = (int)o;
            try
            {
                webReq[i] = (HttpWebRequest)WebRequest.Create(myUrls[i]);
                webReq[i].Proxy = HttpWebRequest.GetSystemWebProxy();
                webReq[i].ServicePoint.Expect100Continue = false;
                webReq[i].AllowAutoRedirect = true;
                webReq[i].KeepAlive = false;
                webReq[i].UserAgent = "Mozilla Firefox";
                webReq[i].Method = "GET";
                webReq[i].Timeout = 5000;
                webReq[i].ContentType = "application/x-www-form-urlencoded";

                // Gimme some sugar!
                webResp[i] = (HttpWebResponse)webReq[i].GetResponse();
                
                Console.WriteLine(sw.Elapsed + " - Finsihed: " + myUrls[i]);
            }
            catch
            {
                // In case of fail, let the user know.
                Console.WriteLine(sw.Elapsed + " - FAILED: " + myUrls[i]);
                resetEvents[i].Set();
                return;
            }

            // All Done. Tell the ThreadPool its finished so it can be re-used.
            resetEvents[i].Set();

        }
    }
}

MY RESULT: 14.968 seconds.

[slide]http://www.imgcentre.com/img/uploads/big/f80ff61bc5.png[/slide]

 

[Hyperz]

using System;
using System.Collections.Generic;
using System.Diagnostics;
using System.IO;
using System.IO.Compression;
using System.Linq;
using System.Net;
using System.Text;
using System.Threading;
using System.Threading.Tasks;

namespace ConsoleApplication1
{
    class Program
    {
        public static Stopwatch sw = new Stopwatch();

        static void Main(string[] args)
        {
            Console.WriteLine("Starting");
            ServicePointManager.Expect100Continue = false;
            Thread.CurrentThread.Priority = ThreadPriority.AboveNormal;
            ServicePointManager.DefaultConnectionLimit = 4;

            RunTest();
            
            Console.Read();
        }

        public static void RunTest()
        {
            string filename = "urls.txt";
            string[] urls = File.ReadAllLines(filename);
            sw.Start();
            foreach (string url in urls) DoRequest(url);
        }

        public static void DoRequest(string url)
        {
            var thread = new Thread(state =>
            {
                var req = (HttpWebRequest)WebRequest.Create(state as String);
                req.AllowAutoRedirect = false;
                req.Timeout = 5000;
                req.Proxy = null;
                req.KeepAlive = false;
                req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");

                try
                {
                    using (var resp = (HttpWebResponse)req.GetResponse())
                    {
                        resp.Close();
                        //req.Abort();
                        Console.WriteLine("{0} - Finished: {1}", sw.Elapsed, resp.ResponseUri.OriginalString);
                    }
                }
                catch
                {
                    req.Abort();
                    Console.WriteLine("{0} - FAILED: {1}", sw.Elapsed, req.RequestUri.OriginalString);
                }
            });

            thread.Priority = ThreadPriority.Highest;
            thread.IsBackground = true;
            thread.Start(url);
        }

        /*public static void DoRequest(string url)
        {
            var req = (HttpWebRequest)WebRequest.Create(url);
            req.AllowAutoRedirect = false;
            req.Timeout = 5000;
            req.Proxy = null;
            req.KeepAlive = false;
            req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");

            var async = req.BeginGetResponse(new AsyncCallback(result =>
            {
                var request = (HttpWebRequest)result.AsyncState;

                try
                {
                    using (var resp = (HttpWebResponse)request.EndGetResponse(result))
                    {
                        resp.Close();
                        request.Abort();
                        Console.WriteLine("{0} - Finished: {1}", sw.Elapsed, resp.ResponseUri.OriginalString);
                    }
                }
                catch
                {
                    request.Abort();
                    Console.WriteLine("{0} - FAILED: {1}", sw.Elapsed, request.RequestUri.OriginalString);
                }
            }), req);

            ThreadPool.RegisterWaitForSingleObject(async.AsyncWaitHandle, new WaitOrTimerCallback((state, timeout) =>
            {
                var request = (HttpWebRequest)state;
                if (timeout)
                {
                    request.Abort();
                    Console.WriteLine("{0} - FAILED: {1}", sw.Elapsed, request.RequestUri.OriginalString);
                }
            }), req, 5000, true);
        }*/
    }
}

RESULT: 5.183 seconds.


Using the propper async way, which is less resource intensive but slow I get around 17 seconds. That would be the part in the code that's commented out.

F# version:

open System
open System.Collections.Generic
open System.Diagnostics
open System.IO
open System.IO.Compression
open System.Linq
open System.Net
open System.Text
open System.Threading
open System.Threading.Tasks

stdout.WriteLine "Starting."

let urls = File.ReadAllLines "urls.txt" |> List.ofArray
let sw = new Stopwatch()

ServicePointManager.Expect100Continue <- false

GC.Collect();

ServicePointManager.DefaultConnectionLimit <- 480
ServicePointManager.MaxServicePoints <- 480

ThreadPool.SetMaxThreads(480, 480) |> ignore
ThreadPool.SetMinThreads(450, 450) |> ignore

let DoRequest url = async {
    try
        let req = WebRequest.Create(url:string) :?> HttpWebRequest
        req.AllowAutoRedirect <- false
        req.Timeout <- 5000
        req.Proxy <- null
        req.KeepAlive <- false
        req.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")
        use! rsp = req.AsyncGetResponse()
        rsp.Close()
        String.Format("{0} - Finished: {1}", sw.Elapsed, url)
        |> stdout.WriteLine
    with _ ->
        String.Format("{0} - FAILED: {1}", sw.Elapsed, url)
        |> stdout.WriteLine
    
    return true }

sw.Start()

urls
|> List.map DoRequest
|> Async.Parallel
|> Async.RunSynchronously
|> ignore

stdout.WriteLine "Done."
stdin.Read() |> ignore

I can't test this one because the timeout isn't working. Should it work it would be around 5 - 7 seconds.

 

[el_jentel1]

I don't know if it's Hyperz pwned Jay or F# pwned C#. Either way, I win.

 

[jayfella]

haha

hang on

 

[Whoo]

Isn't the speed of your internet connection the big unknown factor in this?

 

[Hyperz]

We test both programs on our pc .

 

[jayfella]

using System;
using System.Collections.Generic;
using System.Linq;
using System.Diagnostics;
using System.IO;
using System.Text;

using System.Threading;
using System.Net;

namespace ConsoleApplication1
{

    class Program
    {
        public static string[] myUrls;
        static HttpWebRequest[] webReq;
        static HttpWebResponse[] webResp;
        static ManualResetEvent[] resetEvents;
        static Stopwatch sw = new Stopwatch();

        static void Main(string[] args)
        {
            Console.WriteLine("starting...");
            RunTest();
        }

        public static void RunTest()
        {
            string s = "";
            using (StreamReader rdr = File.OpenText("urls.txt"))
            { s = rdr.ReadToEnd(); }

            myUrls = s.Split('\n');

            ThreadPool.SetMinThreads(512, 512);
            ThreadPool.SetMaxThreads(512, 512);

            DeclareArrays();
            sw.Start();

            for (int i = 0; i < myUrls.Length; i++)
            {
                resetEvents[i] = new ManualResetEvent(false);
                ThreadPool.QueueUserWorkItem(new WaitCallback(DoRequest), (object)i);
            }
            
            Console.Read();
        }

        public static void DeclareArrays()
        {
            webReq = new HttpWebRequest[myUrls.Length];
            webResp = new HttpWebResponse[myUrls.Length];
            resetEvents = new ManualResetEvent[myUrls.Length];
        }

        public static void DoRequest(object o)
        {
            int i = (int)o;
            try
            {
                webReq[i] = (HttpWebRequest)WebRequest.Create(myUrls[i]);
                webReq[i].Proxy = HttpWebRequest.GetSystemWebProxy();
                webReq[i].Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip, deflate");
                webReq[i].ServicePoint.Expect100Continue = false;
                webReq[i].AllowAutoRedirect = false;
                webReq[i].KeepAlive = false;
                webReq[i].UserAgent = "Mozilla Firefox";
                webReq[i].Method = "GET";
                webReq[i].Timeout = 5000;
                webReq[i].ContentType = "application/x-www-form-urlencoded";

                
                webReq[i].AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

                using (webResp[i] = (HttpWebResponse)webReq[i].GetResponse())
                {
                    // webResp[i].Close();
                    Console.WriteLine(sw.Elapsed + " - Finsihed: " + myUrls[i]);
                    resetEvents[i].Set();
                    return;
                }
            }
            catch
            {
                // webResp[i].Close();
                Console.WriteLine(sw.Elapsed + " - FAILED: " + myUrls[i]);
                resetEvents[i].Set();
                return;
            }
            
        }
    }
}

[slide]http://i45.tinypic.com/2mw9so3.png[/slide]

 

[Ronald]

Hyperz you tryhard!

 

[Zeokat]

On jayfella´s code, exists one line not needed using GET method:

webReq[i].ContentType = "application/x-www-form-urlencoded";

Content type is used mostly with POST method... not needed on GET requests.

Anyways... speed results can be confused. All depends on how many timeouts the day of the test, the speed and the load of the server the day of the test.... etc...

Anyways are good pieces of code.

 

[Hyperz]

We tested both apps on both our pc's at the same time.

 

[jayfella]

not really, we ran it on both computers and posted out source in the interest of fairness. The primary objective was trying to get the requests in and out as quickly as possible. After a short period we managed to get so close to each others time that it near on didnt really matter anymore, which was interesting in itself.. 2 different ways of going about the same thing, both with their advantages and dis-advantages.

Btw, that line makes not diff, i just ripped a few lines straight out of one of my programs. So anyway, willing to give it a try...??? Do you have any other alternative method you believe would be any quicker? Its all in the interest of fun and learning - would appreciate yours or anybodys effort

 

[Zeokat]

If the objective is discuss about threading vs asynchronous, i have my doubts that exists an important speed difference.

In the other hand perhaps in resources needed, the difference is big enougth to take care.

If the objective is check dead/alive sites with best speed, for sure my first modification will be change GET method to HEAD method.

Anyways the number of simultaneos requests is a good point to take care of... all will depends on conection speed...

Anyways i have to run some tests... specially since i coded my own HTTP library with sockets and perhaps speed can be increased a little.

For sure are interesting tests, and interesting pieces of code, thanks to both.
I will try to speak lees and add more lines of code, anyways be warned i´m not a professional of computers i self taugth reading books, i´m amateur, so not expect nothing great :P

 

[Hyperz]

HEAD would have zero impact here since the response data doesn't get read.

IMO, threading:
Very fast. Wastes CPU cycles. Eats more memory. Doesn't scale that great.
Async:
Not as fast. Not resource intensive. The way it should be done.


Are you an ex VB coder :p? Why reinvent the wheel (not using HttpWebRequest)?

 

[jayfella]

@Zeokat - we are both self taught - ur contribution will be appreciated - i look forward to seeing your results

 

[Zeokat]

HEAD would have zero impact here since the response data doesn't get read.

To check if a site is dead/alive HEAD method is enougth since headers cotains status code.

Are you an ex VB coder :p? Why reinvent the wheel (not using HttpWebRequest)?

I had to heard that a lot of times, in the past i couldn´t reach a good way to use ThreadPool + HttpWebRequest, always fall in problems (seems that jayfella showed a nice way to do it).

 

[Hyperz]

If the threadpool gives you problems don't use it :p? But at the end of the day the HttpWebRequest class is what you are re-building. The objective is not to check for dead links but to make a request and get the response.

 

[litewarez]

Stupid ass question but where do i store the urls.txt xD
^^ Forget that.

All i get is failed now, during the Exception Section on all urls, any reason why?

 

[Hyperz]

Same folder as the .exe.