Yikes, I ran into a real bummer of an edge case yesterday in one of my older low level handler implementations (for West Wind Web Connection in this case). Basically this handler is a connector for a backend Web framework that creates self contained HTTP output. An ASP.NET Handler captures the full output, and then shoves the result down the ASP.NET Response object pipeline writing out the content into the Response.OutputStream and seperately sending the HttpHeaders in the Response.Headers collection.The headers turned out to be the problem and specifically Http Cookies, which for some reason ended up getting stripped out in some scenarios.

My handler works like this: Basically the HTTP response from the backend app would return a full set of HTTP headers plus the content. The ASP.NET handler would read the headers one at a time and then dump them out via Response.AppendHeader().

But I found that in some situations Set-Cookie headers sent along were simply stripped inside of the Http Handler. After a bunch of back and forth with some folks from Microsoft (thanks Damien and Levi!) I managed to pin this down to a very narrow edge scenario.

It's easiest to demonstrate the problem with a simple example HttpHandler implementation. The following simulates the very much simplified output generation process that fails in my handler. Specifically I have a couple of headers including a Set-Cookie header and some output that gets written into the Response object.

using System.Web;

namespace wwThreads
{

    public class Handler : IHttpHandler
    {
        /* NOTE: 
         * 
         * Run as a web.config set handler (see entry below)
         *
         * Best way is to look at the HTTP Headers in Fiddler
         * or Chrome/FireBug/IE tools and look for the 
         * WWHTREADSID cookie in the outgoing Response headers
         * ( If the cookie is not there you see the problem! )
         */
        public void ProcessRequest(HttpContext context)
        {                        
            HttpRequest request = context.Request;
            HttpResponse response = context.Response;

            // If ClearHeaders is used Set-Cookie header gets removed!
            // if commented header is sent...
            response.ClearHeaders();

            response.ClearContent(); 
                                
            // Demonstrate that other headers make it
            response.AppendHeader("RequestId", "asdasdasd");
            
            // This cookie gets removed when ClearHeaders above is called
            // When ClearHEaders is omitted above the cookie renders
            response.AppendHeader("Set-Cookie", "WWTHREADSID=ThisIsThEValue; path=/");

           // *** This always works, even when explicit 
           //     Set-Cookie above fails and ClearHeaders is called
           //response.Cookies.Add(new HttpCookie("WWTHREADSID", "ThisIsTheValue"));

            response.Write(@"Output was created.<hr/>
                           Check output…");            
        }

        public bool IsReusable
        {
            get { return false; }
        }

    }
}

In order to see the problem behavior this code has to be inside of an HttpHandler, and specifically in a handler defined in web.config with:

      <add name=".ck_handler"
           path="handler.ck"
           verb="*"
           type="wwThreads.Handler"
           preCondition="integratedMode" />

Note: Oddly enough this problem manifests only when configured through web.config, not in an ASHX handler, nor if you paste that same code into an ASPX or .csHtml page.

It does however also manifest in an MVC ActionMethod:

public class CookieBugController : Controller
{
    //
    // GET: /CookieBug/

    public string Index()
    {
        var response = HttpContext.Response;

        // If ClearHeaders is used Set-Cookie header gets removed!
        // if commented header is sent...
        response.ClearHeaders();

        //RemoveHeaders(response);

        response.ClearContent();

        // Demonstrate that other headers make it - this one shows
        response.AppendHeader("RequestId", "0ae133f123d");

        // This cookie gets removed when ClearHeaders above is called
        // When ClearHEaders is omitted above the cookie renders
        response.AppendHeader("Set-Cookie", "WWTHREADSID=ThisIsThEValue; path=/");

        // *** This always works, even when explicit 
        //     Set-Cookie above fails and ClearHeaders is called
        //response.Cookies.Add(new HttpCookie("WWTHREADSIsD", "ThisIsTheValue"));            

        return @"Output was created.<hr/>Check output with Fiddler or HTTP Proxy to see whether cookie was sent.";
    }
}

It appears if the request is not bound to a physical file on disk (ASHX, ASPX etc. pages), then this fails. Bind to a file and it works. For example, if I create a Handler.ck file for the handler the problem goes away.

What's the problem exactly?

The code above simulates the more complex code in my live handler that picks up the HTTP response from the backend application and then peels out the headers and sends them one at a time via Response.AppendHeader. One of the headers in my app can be one or more Set-Cookie.

I found that the Set-Cookie headers were not making it into the Response headers output. Here's the Chrome Http Inspector trace:

Notice, no Set-Cookie header in the Response headers!

Now, running the very same request after removing the call to Response.ClearHeaders() command, the cookie header shows up just fine:

As you might expect it took a while to track this down. At first I thought my backend was not sending the headers but after closer checks I found that indeed the headers were set in the backend HTTP response, and they were indeed getting set via Response.AppendHeader() in the handler code. Yet, no cookie in the output.

In the simulated example the problem is this line:

response.AppendHeader("Set-Cookie", "WWTHREADSID=ThisIsThEValue; path=/");

which in my live code is more dynamic ( ie. AppendHeader(token[0],token[1[]) )as it parses through the headers.

Bizzaro Land: Response.ClearHeaders() causes Cookie to get stripped

Now, here is where it really gets bizarre: The problem occurs only if:

• Response.ClearHeaders() was called before headers are added
• Response.AppendHeader("Set-Cookie","…") was called
• If there's no physical file: web.config handler, or MVC Routed Controller Action
  Never a problem in ASHX, ASPX, csHtml files etc.
• It only occurs if there are WebPages files (.cshtml,.vbhtml) present in the project tree
• Occurs only in IIS and IIS Express but not in the Visual Studio Web Server

So in the code above if you remove the call to ClearHeaders(), the cookie gets set!  Add it back in and the cookie is not there.

If I run the above code in an ASHX handler it works. If I run the same code inside of ASPX page it works. Only in the HttpHandler configured through Web.config does it fail! If I use it inside of an MVC Controller I also get the failure, but it's not very likely that one would use Set-Cookie headers in a controller method I suspect :-) If I remove any .cshtml files from my web application it also works, but of course for MVC applications that's not really an option.

Cue the Twilight Zone Music.

This problem is the weirdest edge case I've ever run into and it took forever to pin it down exactly. In fact on several occasions I posted some code on CodePaste.net only to find out that nobody could reproduce it. No wonder with all the odd dependencies - the final straw was the requirement of ASP.NET WebPages being present. I suspect that this is the real culprit - some sort of pre or post processing that mucks with the headers.

Workarounds

As is often the case the fix for this once you know the problem is not too difficult. The difficulty lies in tracking inconsistencies like this down. Luckily there are a few simple workarounds for the Cookie issue.

Don't use AppendHeader for Cookies

The easiest and obvious solution to this problem is simply not use Response.AppendHeader() to set Cookies. Duh! Under normal circumstances in application level code there's rarely a reason to write out a cookie like this:

response.AppendHeader("Set-Cookie", "WWTHREADSID=ThisIsThEValue; path=/");

but rather create the cookie using the Response.Cookies collection:

response.Cookies.Add(new HttpCookie("WWTHREADSID", "ThisIsTheValue"));

Unfortunately, in my case where I dynamically read headers from the original output and then dynamically  write header key value pairs back  programmatically into the Response.Headers collection, I actually don't look at each header specifically so in my case the cookie is just another header.

My first thought was to simply trap for the Set-Cookie header and then parse out the cookie and create a Cookie object instead. But given that cookies can have a lot of different options this is not exactly trivial, plus I don't really want to fuck around with cookie values which can be notoriously brittle.

Don't use Response.ClearHeaders()

The real mystery in all this is why calling Response.ClearHeaders() prevents a cookie value later written with Response.AppendHeader() to fail. I fired up Reflector and took a quick look at System.Web and HttpResponse.ClearHeaders. There's all sorts of resetting going on but nothing that seems to indicate that headers should be removed later on in the request. The code in ClearHeaders() does access the HttpWorkerRequest, which is the low level interface directly into IIS, and so I suspect it's actually IIS that's stripping the headers and not ASP.NET, but it's hard to know. Somebody from Microsoft and the IIS team would have to comment on that.

In my application it's probably safe to simply skip ClearHeaders() in my handler. The ClearHeaders/ClearContent was mainly for safety but after reviewing my code there really should never be a reason that headers would be set prior to this method firing.

However, if for whatever reason headers do need to be cleared, it's easy enough to manually clear the headers out:

private void RemoveHeaders(HttpResponse response)
{
    List<string> headers = new List<string>();
    foreach (string header in response.Headers)
    {
        headers.Add(header);
    }
    foreach (string header in headers)
    {
        response.Headers.Remove(header);
    }

    response.Cookies.Clear();
}

Now I can replace the call the Response.ClearHeaders() and I don't get the funky side-effects from Response.ClearHeaders().

Summary

I realize this is a total edge case as this affects only the uncommon use of explicitly appending a Cookie Header via Response.AppendHeader. And even then it occurs only if you are running in a project that also contains ASP.NET WebPages files and you're executing a non-file bound request. Lots of odds against that happening. Nevertheless there's a bug here that can bite you, as it did me.

For most application level code this will likely never be a problem. It's so much easier using Response.Cookies.Add() that AppendHeader() for cookie usage should never occur, except in some sort of system component such as mine that directly parses headers into the Http output stream.

IAC, there are workarounds to this should you run into it, although I bet when you do run into it, it'll likely take a bit of time to find the problem or even this post in a search because it's not easily to correlate the problem to the solution. It's quite possible that more than cookies are affected by this behavior. Searching for a solution I read a few other accounts where headers like Referer were mysteriously disappearing, and it's possible that something similar is happening in those cases.

Again, extreme edge case, but I'm writing this up here as documentation for myself and possibly some others that might have run into this.

Resources

• Sample Project that Demonstrates Failure
• My Test Environment:
  IIS 8 on Windows 8 64 bit
  Running a 32 bit AppPool
  Running ASP.NET 4.0 (not 4.5)