AboutCode.net - Async WebRequest using LINQ syntax

Thoughts on Software by Andrew Davey

Monday, January 14, 2008

Async WebRequest using LINQ syntax

Wes Dyer's paper http://blogs.msdn.com/wesdyer/archive/2008/01/11/the-marvels-of-monads.aspx has a great introduction to using Monads in C# and specifically how to create a continuation Monad that leverages the LINQ syntax.

After a lot of re-reading and coding, I managed to construct an asynchronous web request program.

UPDATE: Added Select<T> method for the monad so "let" syntax works.

class Program
{
    static void Main(string[] args)
    {
        var requests = new[]
        {
            WebRequest.Create("http://www.google.com/"),
            WebRequest.Create("http://www.yahoo.com/"),
            WebRequest.Create("http://channel9.msdn.com/")
        };
        var pages = from request in requests
            select
                from response in request.GetResponseAsync()
                let stream = response.GetResponseStream()
                from html in stream.ReadToEndAsync()
                select new { html, response };

        foreach (var page in pages)
        {
            page(d =>
            {
                Console.WriteLine(d.response.ResponseUri.ToString());
                Console.WriteLine(d.html.Substring(0, 40));
                Console.WriteLine();
            });
        }

Console.ReadKey();
}
}

The actual web requests are deferred until the for loop invokes each "page" method. If you run the example a few times, you see that the results come back in a non-deterministic order, as expected with concurrent asynchronous operations. In addition, the web response stream is read asynchronously. So the main thread is never blocked - in fact that's why we need the Console.ReadKey() at the end, to stop the process ending before the results come back!

I had to change the type of the continuations from what Wes used. Instead of Func<T, Answer> I am just using Action<T>. The following code implements the continuation monad and the additional WebRequest and Stream extension methods.

delegate void K<T>(Action<T> c);

static class Continuations
{
    public static K<T> ToContinuation<T>(this T value)
    {
        return c => c(value);
    }

    public static K<U> SelectMany<T, U>(this K<T> m, Func<T, K<U>> k)
    {
        return c => m(x => k(x)(c));
    }

    public static K<V> SelectMany<T, U, V>(this K<T> m, Func<T, K<U>> k, Func<T, U, V> s)
    {
        return m.SelectMany(x => k(x).SelectMany(y => s(x, y).ToContinuation<V>()));
    }

    public static K<U> Select<U, T>(this K<T> m, Func<T, U> k)
    {
        return c => m(x => c(k(x)));
    }

    public static K<WebResponse> GetResponseAsync(this WebRequest request)
    {
        return send => request.BeginGetResponse(result =>
        {
            var response = request.EndGetResponse(result);
            send(response);
        }, null);
    }

    public static K<string> ReadToEndAsync(this Stream stream)
    {
        return send =>
        {
            byte[] buffer = new byte[1024];
            StringBuilder sb = new StringBuilder();
            Func<IAsyncResult> readChunk = null;
            readChunk = () => stream.BeginRead(buffer, 0, 1024, result =>
            {
                int read = stream.EndRead(result);
                if (read > 0)
                {
                    sb.Append(Encoding.UTF8.GetString(buffer, 0, read));
                    readChunk();
                }
                else
                {
                    stream.Dispose();
                    send(sb.ToString());
                }
            }, null);
            readChunk();
        };
    }

}

I imagine we will see something like this come out of the Volta project soon. Hopefully, it will form a generic asynchronous library that we can all use with anything that implements the Begin/End Invoke pattern.

Monday, January 14, 2008 1:22:13 PM (GMT Standard Time, UTC+00:00)

Name
E-mail
Home page

	Remember Me
Comment (HTML not allowed)
Enter the code shown (prevents robots):