Rick Strahl's Weblog  

Wind, waves, code and everything in between...
.NET • C# • Markdown • JavaScript • Angular
Contact   •   Articles   •   Products   •   Support   •   Search
Ad-free experience sponsored by:
ASPOSE - the market leader of .NET and Java APIs for file formats – natively work with DOCX, XLSX, PPT, PDF, images and more

WebClient and GetWebResponse not firing on Async Requests


:P
On this page:

Here’s a little oddity I ran into today: When you’re using the good old simple WebClient class you can subclass WebClient and capture the HttpWebResponse object. This is useful because WebClient doesn’t expose any of the important HTTP results like Http status code or headers. WebClient is meant to be ultra basic, but by capturing the Response you actually get most of the features you need of a full HTTP client. The most important ones that you’ll typically want to look at are the Http Status code and the Response headers. WebClient is just a high level wrapper around HttpWebRequest/HttpWebResponse and using a couple of overrides you can actually capture both low level interfaces without having to resort to using HttpWebRequest/Response which is considerably more work to use.

Recently I needed to build a set of HTTP utilities because I had some problems with the all async HttpClient – specifically some thread abort exceptions that weren’t giving me any more info on the error – and I decided to just create a small wrapper to make JsonRequest calls.

As part of the implementation I subclassed WebClient like this in order to capture the HttpWebResponse object:

/// <summary>
/// Customized version of WebClient that provides access
/// to the Response object so we can read result data 
/// from the Response.
/// </summary>
internal class HttpUtilsWebClient : WebClient
{
    internal HttpWebResponse Response { get; set; }

    protected override WebResponse GetWebResponse(WebRequest request)
    {
        Response = base.GetWebResponse(request) as HttpWebResponse;
        return Response;
    }
        
}

and then created some helper methods that wrap up a JsonRequest<T> routine that basically let you retrieve and convert Json (you can check out the full code for this on GitHub for the HttpUtils class in Westwind.Utilities.

public static TResultType JsonRequest<TResultType>(HttpRequestSettings settings)
{
    var client = new HttpUtilsWebClient();

    if (settings.Credentials != null)
        client.Credentials = settings.Credentials;

    if (settings.Proxy != null)
        client.Proxy = settings.Proxy;
            
    client.Headers.Add("Accept", "application/json");

    if (settings.Headers != null)
    {
        foreach (var header in settings.Headers)
        {
            client.Headers[header.Key] = header.Value;
        }
    }

    string jsonResult;

    if (settings.HttpVerb == "GET")
        jsonResult = client.DownloadString(settings.Url);
    else
    {
        if (!string.IsNullOrEmpty(settings.ContentType))
            client.Headers["Content-type"] = settings.ContentType;
        else
            client.Headers["Content-type"] = "application/json";

        if (!settings.IsRawData)
            settings.CapturedRequestContent = JsonSerializationUtils.Serialize(settings.Content, throwExceptions: true);
        else
            settings.CapturedRequestContent = settings.Content as string;

        jsonResult = client.UploadString(settings.Url, settings.HttpVerb, settings.CapturedRequestContent);

        if (jsonResult == null)
            return default(TResultType);
    }

    settings.CapturedResponseContent = jsonResult;
    settings.Response = client.Response;
            
    return (TResultType) JsonSerializationUtils.Deserialize(jsonResult, typeof (TResultType), true);
}

Inside of that code I basically create custom instance of the HttpUtilsWebClient and then capture the response when done, to pass back to the caller as part of a settings object that is passed in initially:

settings.Response = client.Response;

When running the standard synchronous version this works perfectly fine.

Using the above code I can do stuff like this:

[TestMethod]
public void JsonRequestPostAsyncTest()
{
    var postSnippet = new CodeSnippet()
    {
        UserId = "Bogus",
        Code = "string.Format('Hello World, I will own you!');",
        Comment = "World domination imminent"
    };           

    var settings = new HttpRequestSettings()
    {
        Url = "http://codepaste.net/recent?format=json",
        Content = postSnippet,
        HttpVerb = "POST"
    };
            
    var snippets = HttpUtils.JsonRequest<List<CodeSnippet>>(settings);

    Assert.IsNotNull(snippets);
    Assert.IsTrue(settings.ResponseStatusCode == System.Net.HttpStatusCode.OK);
    Assert.IsTrue(snippets.Count > 0);

    Console.WriteLine(snippets.Count);
    Console.WriteLine(settings.CapturedRequestContent);
    Console.WriteLine();
    Console.WriteLine(settings.CapturedResponseContent);

    foreach (var snippet in snippets)
    {
        if (string.IsNullOrEmpty(snippet.Code))
            continue;
        Console.WriteLine(snippet.Code.Substring(0, Math.Min(snippet.Code.Length, 200)));
        Console.WriteLine("--");
    }
Console.WriteLine("Status Code: " + settings.Response.StatusCode);
foreach (var header in settings.Response.Headers) { Console.WriteLine(header + ": " + settings.Response.Headers[header.ToString()]); } }
Note that I can look at the Response object settings. I can get the HTTP status code and look at the settings.Response.Headers if I need to.

Async Fail?

I also created an async version which is pretty much identical to the sync version except for the async and await semantics (and note how it’s not so easy to reuse existing code unless you can factor the pieces in great detail so this method is practically a copy of the first):

public static async Task<TResultType> JsonRequestAsync<TResultType>(HttpRequestSettings settings)
{
    var client = new HttpUtilsWebClient();
    
    if (settings.Credentials != null)
        client.Credentials = settings.Credentials;

    if (settings.Proxy != null)
        client.Proxy = settings.Proxy;

    client.Headers.Add("Accept", "application/json");

    if (settings.Headers != null)
    {
        foreach (var header in settings.Headers)
        {
            client.Headers[header.Key] = header.Value;
        }
    }

    string jsonResult;
    if (settings.HttpVerb == "GET")
        jsonResult = await client.DownloadStringTaskAsync(settings.Url);                
    else
    {
        if (!string.IsNullOrEmpty(settings.ContentType))
            client.Headers["Content-type"] = settings.ContentType;
        else
            client.Headers["Content-type"] = "application/json";

        if (!settings.IsRawData)
            settings.CapturedRequestContent = JsonSerializationUtils.Serialize(settings.Content, throwExceptions: true);
        else
            settings.CapturedRequestContent = settings.Content as string;

        jsonResult = await client.UploadStringTaskAsync(settings.Url, settings.HttpVerb, settings.CapturedRequestContent);

        if (jsonResult == null)
            return default(TResultType);
    }

    settings.CapturedResponseContent = jsonResult;
    settings.Response = client.Response;

    return (TResultType) JsonSerializationUtils.Deserialize(jsonResult, typeof (TResultType), true);
    //return JsonConvert.Deserialize<TResultType>(jsonResult);
}

The call to this method is the same except for the await keyword and Async method called:

var snippets = await HttpUtils.JsonRequestAsync<List<CodeSnippet>>(settings);

This works fine for the HTTP retrieval and parsing, but unfortunately you don’t get the settings.Response instance and therefore no access to the Http Status code or header. The test code from above fails when trying to read the Status code because .Response is null. Argh.

When you’re using the async versions of WebClient (like DownloadStringAsyncTask()) the Response object is never assigned because the overriden GetResponse() method is never fired in the overload as it is in the synchronous call for client.UploadStringAsyncTask().

It turns out that there’s another overload of GetWebResponse() (thanks to Damien Edwards for pointing that out and making me feel like I missed the obvious now :-)) that takes an IAsyncResult input. The reason I missed this originally is that I thought pertained to the ‘old’ async interfaces and so dismissed it. Only when Damien pointed it out did I give that overload a try – and it works!

The updated HttpUtilsWebClient looks like this:

public class HttpUtilsWebClient : WebClient
{
    internal HttpWebResponse Response { get; set; }

    protected override WebResponse GetWebResponse(WebRequest request)
    {
        Response = base.GetWebResponse(request) as HttpWebResponse;
        return Response;
    }

    protected override WebResponse GetWebResponse(WebRequest request, System.IAsyncResult result)
    {
        Response = base.GetWebResponse(request, result) as HttpWebResponse;
        return Response;
    }
        
}

Now, with this code in place the async tests that access the Response object to get the Http Status code and access the Response HTTP headers work fine. Yay!

With Access to Response, WebClient becomes a lot more useful!

Getting access to the Response object makes WebClient a heck of a lot more useful above and beyond its simple interface. And its simple interface really is a bonus if you just need quick and dirty HTTP access in an app. And it’s just built into the standard .NET libraries – no additional dependencies required and it also works with pre-4.5 versions of .NET.

This is great for simple and easy HTTP access, but also for simple wrappers like the one I described above. Having a simple helper to make JSON calls, plus the ability to capture input and output data is going to make some testing and debugging scenarios much easier…



West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2021