WebRequest and SSL Connection Problems

April 17, 2007 • from Maui, Hawaii • 4 comments

On this page:

Over the weekend I got a message from Chris on the message board. He ran into problems with my Credit Class processing classes for .NET that I posted some time ago. Basically what happens is this:

You connect to a secure site with WebRequest and the first request works perfectly fine. Then a second request is done quickly thereafter and the second request will always fail. A typical exception in this scenario might be:

A first chance exception of type 'System.ObjectDisposedException' occurred in System.dll
Additional information: Cannot access a disposed object.

Now this is happening even though a completely new instance of a WebRequest object is created for each Http request. Http after all is stateless right <s>...

Lest you think this is just my scrappy code (I have a high level wwHTTP wrapper around WebRequest) <s>- I've run into this same situation with Web Service Proxies. Anything that uses WebRequest under the covers is possibly prone to this issue.

I've seen this issues with various servers but it seems to occur mostly with SSL transactions and more often than not when running through some sort of proxy server. For example I can make this fail by hooking up to Fiddler and running two requests. The second request will always fail.

The problem is that WebRequest uses a ServicePoint under the covers with a global ServicePointManager that is responsible for caching and dishing HTTP connections. Now https: requests are often chached agressively because SSL connections are fairly expensive to set up because there's a fair amount of back and forth messaging that occurs for the SSL handshakes. So ServicePointManager holds on to connections and tries to dish another one if you hit the same server in quick succession.

I've never run into this problem with non-SSL requests but I've seen this on a few occasions myself with HTTPS requests and always in scenarios where requests are repeated in quick succession with new instances of WebRequest objects (in this case for unrelated requests - ie. it's not a loop, but successive Credit Card Processing requests).

In any case, regardless of what the problem is you might run into this and when you do there's a fairly easy fix which is to turn of Http Keep-Alives for the http request. If you're making a single server request it's very easy to just tell WebRequest to close the connection:

HttpWebRequest request = HttpWebRequest.Create("https://gateway.merchantplus.com/...");

request.KeepAlive = false;

...

WebResponse response = request.GetResponse();

This takes care of the problem. You'll want to be careful not to use this code though on everything - if you make multiple server calls you certainly will want to take advantage of keep alives on SSL calls if possible. I suspect the problem is that the server might be forcing the connection closed even though the client asks for keep-alives (which is often done by proxies) and that's what actually causes the problem with the ServicePointManager.

On a slightly different topic - Chris sent me a complete network trace of his request which was a great help in tracking this down. The amount of information available on these traces is a tremendous help. You can find out how to set up a network trace in this MSDN article.

The Voices of Reason

Andy olson
April 17, 2007

# re: WebRequest and SSL Connection Problems

First of all, Great white paper on “Retrieving HTTP content in .Net”! There are several posts around that seem to tell half the story, while yours seems to touch on a greater majority of the issues. Thank you.

This post of the HttpWebRequest his very close to home on a problem that we are currently trying to deal with, and I’m wondering if they’re related, or if you might have some light to shed on it.

We have a large site that we’re in the process of converting from Classic asp to ASP.NET 2.0, and in this particular situation, we need to have the new .net code request the html output from a couple of the legacy ASP pages to later be stiched together and saved as an archive. However we encounter a System.Net.WebException with a description of “The remote server returned an error: (401) Unauthorized.”

While this would seem to be a simple NetworkCredential problem, we’ve verified that the account and password were passing is valid, AND we only experience the problem when we request the page using the fully qualified DNS name. If we request the site using the server name (local network), we do not get the error (using same code and same credentials).

To add to the strangeness, this archive process requires that we also request a few of the converted ASP.NET 2.0 pages to add to the stitched together archive file, and we only get the WebException on those if IIS has Integrated Windows Authentication enabled AND we request the page using the fully qualified DNS name and/or if we hit the site over SSL. If we request using the server name directly it also works fine regardless of the IIS security configuration.

Here's our code:

public static byte[] Execute(string url, int timeout, string userName, string domain, string password)
{
    if(String.IsNullOrEmpty(url))
    {
        throw new ArgumentNullException("url");
    }

    HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(url);
    request.Timeout = timeout;
    request.UserAgent = "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; InfoPath.1; .NET CLR 2.0.50727; .NET CLR 1.1.4322)";
    if(!String.IsNullOrEmpty(userName))
    {
        request.Credentials = new NetworkCredential(userName, password, domain);
    }

    WebResponse response = null;
    Stream stream = null;
    try
    {
        response = request.GetResponse();
        stream = response.GetResponseStream();

        List<byte> bytes = new List<byte>();
        int b = -1;
        while((b = stream.ReadByte()) != -1)
        {
            bytes.Add((byte)b);
        }

        return bytes.ToArray();
    }
    finally
    {
        if(response != null)
        {
            response.Close();
        }
        if(stream != null)
        {
            stream.Close();
            stream.Dispose();
        }
    }
}

.

Any thoughts?

Keep up the great posts!
--
Andy Olson
andy.j.olson(AT)saic.com

Andy Olson
April 17, 2007

# re: WebRequest and SSL Connection Problems

As a follow-up... We just figured out the correct configuration. By adding the following lines of code where we were setting the credentials, the problem went away.

CredentialCache cache = new CredentialCache();
cache.Add(request.RequestUri, "NTLM", new NetworkCredential(userName, password, domain));
cache.Add(request.RequestUri, "Digest", new NetworkCredential(userName, password, domain));
request.Credentials = cache;

We had tried several different combinations of credentials, but the killer was, in most of our tests we were setting a CredentialCache with "Negotiated" as well, which ultimately was creating the 401 error.

Thanks anyway, and keep the good info flowing!

Rick Strahl's Web Log
June 23, 2007

# Rick Strahl's Web Log

Lab
January 05, 2012

# re: WebRequest and SSL Connection Problems

Thank you very much!

The SSL connection cache drive me crazy for a while.

my app is doing different connection sometime with cert check and sometime not...

the result, my app was doing some funny things.

Thank you very much!

Rick Strahl's Weblog