Contact   •   Products   •   Search

Rick Strahl's Web Log

Wind, waves, code and everything in between...
ASP.NET • C# • HTML5 • JavaScript • AngularJs

ASP.NET GZip Encoding Caveats


GZip encoding in ASP.NET is pretty easy to accomplish using the built-in GZipStream and DeflateStream classes and applying them to the Response.Filter property.  While applying GZip and Deflate behavior is pretty easy there are a few caveats that you have watch out for as I found out today for myself with an application that was throwing up some garbage data. But before looking at caveats let’s review GZip implementation for ASP.NET.

ASP.NET GZip/Deflate Basics

Response filters basically are applied to the Response.OutputStream and transform it as data is written to it through the ASP.NET Response object. So a Response.Write eventually gets written into the output stream which if a filter is also written through the filter stream’s interface. To perform the actual GZip (and Deflate) encoding typically used by Web pages .NET includes the GZipStream and DeflateStream stream classes which can be readily assigned to the Repsonse.OutputStream.

With these two stream classes in place it’s almost trivially easy to create a couple of reusable methods that allow you to compress your HTTP output. In my standard WebUtils utility class (from the West Wind West Wind Web Toolkit) created two static utility methods – IsGZipSupported and GZipEncodePage – that check whether the client supports GZip encoding and then actually encodes the current output (note that although the method includes ‘Page’ in its name this code will work with any ASP.NET output).

/// <summary>
/// Determines if GZip is supported
/// </summary>
/// <returns></returns>
public static bool IsGZipSupported()
{
    string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
    if (!string.IsNullOrEmpty(AcceptEncoding) &&
            (AcceptEncoding.Contains("gzip") || AcceptEncoding.Contains("deflate")))
        return true;
    return false;
}

/// <summary>
/// Sets up the current page or handler to use GZip through a Response.Filter
/// IMPORTANT:  
/// You have to call this method before any output is generated!
/// </summary>
public static void GZipEncodePage()
{
    HttpResponse Response = HttpContext.Current.Response;

    if (IsGZipSupported())
    {
        string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];

        if (AcceptEncoding.Contains("deflate"))
        {
            Response.Filter = new System.IO.Compression.DeflateStream(Response.Filter,
                                        System.IO.Compression.CompressionMode.Compress);
            Response.Headers.Remove("Content-Encoding");
            Response.AppendHeader("Content-Encoding", "deflate");
        }
        else
        {
            Response.Filter = new System.IO.Compression.GZipStream(Response.Filter,
                                        System.IO.Compression.CompressionMode.Compress);
            Response.Headers.Remove("Content-Encoding");
            Response.AppendHeader("Content-Encoding", "gzip");
        }

    }
}

As you can see the actual assignment of the Filter is as simple as:

Response.Filter = new DeflateStream(Response.Filter, System.IO.Compression.CompressionMode.Compress);

which applies the filter to the OutputStream. You also need to ensure that your response reflects the new GZip or Deflate encoding and ensure that any pages that are cached in Proxy servers can differentiate between pages that were encoded with the various different encodings (or no encoding).

To use this utility function now is trivially easy: In any ASP.NET code that wants to compress its Response output you simply use:

protected void Page_Load(object sender, EventArgs e)
{            
    WebUtils.GZipEncodePage();

    Entry = WebLogFactory.GetEntry();

    var entries = Entry.GetLastEntries(App.Configuration.ShowEntryCount, "pk,Title,SafeTitle,Body,Entered,Feedback,Location,ShowTopAd", "TEntries");
    if (entries == null)
        throw new ApplicationException("Couldn't load WebLog Entries: " + Entry.ErrorMessage);

    this.repEntries.DataSource = entries;
    this.repEntries.DataBind();

}

Here I use an ASP.NET page, but the above WebUtils.GZipEncode() method call will work in any ASP.NET application type including HTTP Handlers. The only requirement is that the filter needs to be applied before any other output is sent to the OutputStream. For example, in my CallbackHandler service implementation by default output over a certain size is GZip encoded. The output that is generated is JSON or XML and if the output is over 5k in size I apply WebUtils.GZipEncode():

if (sbOutput.Length > GZIP_ENCODE_TRESHOLD)
    WebUtils.GZipEncodePage();

Response.ContentType = ControlResources.STR_JsonContentType;
HttpContext.Current.Response.Write(sbOutput.ToString());

Ok, so you probably get the idea: Encoding GZip/Deflate content is pretty easy.

Hold on there Hoss –Watch your Caching

Or is it? There are a few caveats that you need to watch out for when dealing with GZip content. The fist issue is that you need to deal with the fact that some clients don’t support GZip or Deflate content. Most modern browsers support it, but if you have a programmatic Http client accessing your content GZip/Deflate support is by no means guaranteed. For example, WinInet Http clients don’t support GZip out of the box – it has to be explicitly implemented. Other low level HTTP clients on other platforms too don’t support GZip out of the box.

The problem is that your application, your Web Server and Proxy Servers on the Internet might be caching your generated content. If you return content with GZip once and then again without, either caching is not applied or worse the wrong type of content is returned back to the client from a cache or proxy. The result is an unreadable response for *some clients* which is also very hard to debug and fix once in production.

You already saw the issue of Proxy servers addressed in the GZipEncodePage() function:

// Allow proxy servers to cache encoded and unencoded versions separately
Response.AppendHeader("Vary", "Content-Encoding");

This ensures that any Proxy servers also check for the Content-Encoding HTTP Header to cache their content – not just the URL.

The same thing applies if you do OutputCaching in your own ASP.NET code. If you generate output for GZip on an OutputCached page the GZipped content will be cached (either by ASP.NET’s cache or in some cases by the IIS Kernel Cache). But what if the next client doesn’t support GZip? She’ll get served a cached GZip page that won’t decode and she’ll get a page full of garbage. Wholly undesirable. To fix this you need to add some custom OutputCache rules by way of the GetVaryByCustom() HttpApplication method in your global_ASAX file:

public override string GetVaryByCustomString(HttpContext context, string custom)
{
    // Override Caching for compression
    if (custom == "GZIP")
    {
        string acceptEncoding = HttpContext.Current.Response.Headers["Content-Encoding"];
        
        if (string.IsNullOrEmpty(acceptEncoding))
            return "";
        else if (acceptEncoding.Contains("gzip"))
            return "GZIP";
        else if (acceptEncoding.Contains("deflate"))
            return "DEFLATE";
        
        return "";
    }
    
    return base.GetVaryByCustomString(context, custom);
}

In a page that use Output caching you then specify:

<%@ OutputCache Duration="180" VaryByParam="none" VaryByCustom="GZIP" %>

To use that custom rule.

It’s all Fun and Games until ASP.NET throws an Error

Ok, so you’re up and running with GZip, you have your caching squared away and your pages that you are applying it to are jamming along. Then BOOM, something strange happens and you get a lovely garbled page that look like this:

GarbledOutput

Lovely isn’t it?

What’s happened here is that I have WebUtils.GZipEncode() applied to my page, but there’s an error in the page. The error falls back to the ASP.NET error handler and the error handler removes all existing output (good) and removes all the custom HTTP headers I’ve set manually (usually good, but very bad here). Since I applied the Response.Filter (via GZipEncode) the output is now GZip encoded, but ASP.NET has removed my Content-Encoding header, so the browser receives the GZip encoded content without a notification that it is encoded as GZip. The result is binary output. Here’s what Fiddler says about the raw HTTP header output when an error occurs when GZip encoding was applied:

HTTP/1.1 500 Internal Server Error
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Sat, 30 Apr 2011 22:21:08 GMT
Content-Length: 2138
Connection: close

�`I�%&/m�{J�J��t��` … binary output striped here

Notice: no Content-Encoding header and that’s why we’re seeing this garbage. ASP.NET has stripped the Content-Encoding header but left our filter intact.

So how do we fix this? In my applications I typically have a global Application_Error handler set up and in this case I’ve been using that. One thing that you can do in the Application_Error handler is explicitly clear out the Response.Filter and set it to null at the top:

protected void Application_Error(object sender, EventArgs e)
{
        // Remove any special filtering especially GZip filtering
        Response.Filter = null;
}

And voila I get my Yellow Screen of Death or my custom generated error output back via uncompressed content. BTW, the same is true for Page level errors handled in Page_Error or ASP.NET MVC Error handling methods in a controller.

Another and possibly even better solution is to check whether a filter is attached just before the headers are sent to the client as pointed out by Adam Schroeder in the comments:

 protected void Application_PreSendRequestHeaders()
{
    // ensure that if GZip/Deflate Encoding is applied that headers are set
    // also works when error occurs if filters are still active
    HttpResponse response = HttpContext.Current.Response;
    if (response.Filter is GZipStream && response.Headers["Content-encoding"] != "gzip")
        response.AppendHeader("Content-encoding", "gzip");
    else if (response.Filter is DeflateStream && response.Headers["Content-encoding"] != "deflate")
        response.AppendHeader("Content-encoding", "deflate");
}

This uses the Application_PreSendRequestHeaders() pipeline event to check for compression encoding in a filter and adjusts the content accordingly. This is actually a better solution since this is generic – it’ll work regardless of how the content is cleaned up. For example, an error Response.Redirect() or short error display might get changed and the filter not cleared and this code actually handles that. Sweet, thanks Adam.

It’s unfortunate that ASP.NET doesn’t natively clear out Response.Filters when an error occurs just as it clears the Response and Headers. I can’t see where leaving a Filter in place in an error situation would make any sense, but hey - this is what it is and it’s easy enough to fix as long as you know where to look. Riiiight!

IIS and GZip

I should also mention that IIS 7 includes good support for compression natively. If you can defer encoding to let IIS perform it for you rather than doing it in your code by all means you should do it! Especially any static or semi-dynamic content that can be made static should be using IIS built-in compression. Dynamic caching is also supported but is a bit more tricky to judge in terms of performance and footprint. John Forsyth has a great article on the benefits and drawbacks of IIS 7 compression which gives some detailed performance comparisons and impact reviews. I’ll post another entry next with some more info on IIS compression since information on it seems to be a bit hard to come by.

Related Content

Make Donation
Posted in ASP.NET   IIS7  


Feedback for this Post

 
# re: ASP.NET GZip Encoding Caveats
by Adam Schroder May 02, 2011 @ 7:20am
I was working on this same issue today and solved it like this (opposite of your solution). I think that your solution is probably better. It really is a shame it doesn't do this automatically.


   protected void Application_PreSendRequestHeaders()
    {
        HttpResponse response = HttpContext.Current.Response;
        if (response.Filter is GZipStream && response.Headers["Content-encoding"] != "gzip")
            response.AppendHeader("Content-encoding", "gzip");
        else if (response.Filter is DeflateStream && response.Headers["Content-encoding"] != "deflate")
            response.AppendHeader("Content-encoding", "deflate");
    }
 
# re: ASP.NET GZip Encoding Caveats
by Rick Strahl May 02, 2011 @ 3:16pm
@Adam - actually that's a pretty sweet way to add the headers and that's actually more reliable since it captures all scenarios where the response may be cleared out inadvertantly.

For example I have various service implementation where handlers manage error trapping and error return and there too I know I've forgotten in the past to clear out filters and the above would capture that.
# re: ASP.NET GZip Encoding Caveats
by lonely soul May 06, 2011 @ 8:48pm
great article as usual Rick.
# re: ASP.NET GZip Encoding Caveats
by Matt May 09, 2011 @ 7:00am
I think you need to detect IE6 clients also and not serve gzipped content to them. My understanding is that IE6 (prior to SP2?) will report that it can accept gzip content but will screw up the decoding of the compressed content under various, seemingly random, scenarios.
# re: ASP.NET GZip Encoding Caveats
by Andy May 09, 2011 @ 4:56pm
Adam's solution is very reliable but with one caveat... The GZipStream must be the last filter in the Response.Filter stream chain. (Though I can't think if any situation where it wouldn't be...) But it's possible that streams are chained after the GZipStream to the Response.Filter, therefore getting the Response.Filter before sending headers will return whichever stream was last applied.

For example consider:
Response.Filter = new System.IO.Compression.GZipStream(Response.Filter,
     System.IO.Compression.CompressionMode.Compress);

... later in the same request execution ...
Response.Filter = new MyFilter2(Response.Filter);

then...
(Response.Filter is GZipStream) == false;
# re: ASP.NET GZip Encoding Caveats
by vimal May 10, 2011 @ 5:02am
Very nice article
# re: ASP.NET GZip Encoding Caveats
by Cbrcoder May 16, 2011 @ 10:36pm
Don't you think this is a webserver concern and best not to use it in code ? Just make a comparison of performance difference between IIS 7.5 and .NET version of compression libraries, there is atleast 10x difference in speed. It's best for programmers to not mess with what is a SERVER concern.
# re: ASP.NET GZip Encoding Caveats
by Rick Strahl May 17, 2011 @ 10:15am
@CBrCoder - I don't agree with you. IIS's dynamic compression gives you no control over what gets compressed - it's all or nothing. In my apps where I use compression of dynamic content I really just want a few things compressed not everything. Additionally a lot of the overhead of compression can be negated by properly using Caching in your applications.

FWIW, I posted a second blog entry that talks about built-in compression - choices are good :-)
# re: ASP.NET GZip Encoding Caveats
by novi4ok July 06, 2011 @ 11:48pm
Hello!
I have one question about this code...
I have some errors, because i cant find what i need to write in
using
line...
for example my Visual Studio underline this code
Entry = WebLogFactory.GetEntry();
. I dont know where is WebLogFactory, in which using it is?
# re: ASP.NET GZip Encoding Caveats
by Andr July 12, 2011 @ 2:00pm
Rick, you are very cool guy. You as usual save my hours and even days of dumb work by your great articles.
# re: ASP.NET GZip Encoding Caveats
by kralcafe July 28, 2011 @ 3:55pm
I think you need to detect IE6 clients also and not serve gzipped content to them.
# re: ASP.NET GZip Encoding Caveats
by Wynand Murray September 01, 2011 @ 3:38am
Hi Rick, thanks for a great post, this has REALLY brought down my site's traffic in size!
I did find one error in your "IsGZipSupported" function though:

I am referring to the following line:

if (!string.IsNullOrEmpty(AcceptEncoding) && (AcceptEncoding.Contains("gzip") || AcceptEncoding.Contains("deflate")))

The exception occurs when the client browser does NOT return an Encoding Header.
Upon checking if the "AcceptEncoding" value (Which will be null) "Contains" some text, an Object Reference exception is thrown.

I fixed this by checking if "AcceptEncoding" is null WITHOUT using the "&&" operator to search for the text value inside it. I only use the "Contains" method to search for values after "AcceptEncoding" has been identified as having a value.

This was strange to me since I didn't think that testing for a "Null" AND a value with the "&&" operator would result in an exception in this case.

Thanks for all your great posts, please keep them coming :)

Regards,

Wynand Murray.
# re: ASP.NET GZip Encoding Caveats
by Rick Strahl September 01, 2011 @ 9:31am
@Wyand - I'm not sure why that should fail. If the value is null the condition !string.IsNullOrEmpty() will fail and should terminate the if expression via short circuiting - the && should never evaluate. If it indeed fails then there'd be a serious bug in the .NET compiler not properly short circuiting which would break lots of code. You sure there's not something else going on when this fails?
# re: ASP.NET GZip Encoding Caveats
by blueray September 28, 2011 @ 3:36am
Hi,

excellent article, thanks.

I got problem though. I used the workaround for appending lost content-encoding header by Adam Schroder , but HttpContext.Current is always null inside Application_PreSendRequestHeaders, thus on exception I receive that scrambled text. Any ideas why HttpContext.Current is null ?
# re: ASP.NET GZip Encoding Caveats
by DotNetWise March 24, 2012 @ 3:22pm
Weird enough, ASP.NET MVC 4 Web API ignores IIS gzip compression.
Any ideas how to turn it on in that case too?
# re: ASP.NET GZip Encoding Caveats
by Rick Strahl March 24, 2012 @ 7:26pm
@DotNetWise - what gets ignored exactly? Are you talking about IIS dynamic compression, static compression, ASP.NET based filter GZip compression?
# re: ASP.NET GZip Encoding Caveats
by DotNetWise March 26, 2012 @ 2:01am
IIS' dynamic compression is ignored on ASP.NET MVC 4 for Actions who return JsonResult or application/json; charset=UTF-8; although the web.config explicitely enables dynamic compression to *.*

<httpCompression sendCacheHeaders="true" directory="c:\temp" minFileSizeForComp="10" noCompressionForProxies="false">
      <clear/>
      <scheme name="gzip" dll="%Windir%\system32\inetsrv\gzip.dll" staticCompressionLevel="9" />
      <dynamicTypes>
        <clear/>
        <add mimeType="text/*" enabled="true" />
        <add mimeType="message/*" enabled="true" />
        <add mimeType="application/x-javascript" enabled="true" />
        <add mimeType="application/json" enabled="true" />
        <add mimeType="application/json; charset=UTF-8" enabled="true"   />
        <add mimeType="*/*" enabled="true" />
      </dynamicTypes>
      <staticTypes>
        <add mimeType="text/*" enabled="true" />
        <add mimeType="message/*" enabled="true" />
        <add mimeType="application/x-javascript" enabled="true" />
        <add mimeType="application/atom+xml" enabled="true" />
        <add mimeType="application/xaml+xml" enabled="true" />
        <add mimeType="application/javascript" enabled="true" />
        <add mimeType="application/javascript; charset=UTF-8" enabled="true" />
        <add mimeType="application/json" enabled="true" />
        <add mimeType="application/json; charset=UTF-8" enabled="true" />
        <add mimeType="*/*" enabled="true" />
      </staticTypes>
    </httpCompression>
# re: ASP.NET GZip Encoding Caveats
by James Stoertz April 12, 2012 @ 8:27am
Rick, you are brilliant. BRILLIANT!
We had the "garbled" problem (on errors) for months and couldn't resolve it. Couldn't even search for it because of the binary screen.
# re: ASP.NET GZip Encoding Caveats
by Lachlan July 05, 2012 @ 2:41pm
Hi, good article thanks.

I tried throwing an error after setting the filter without doing anything clever, and it seemed to strip the filter itself without any intervention from me?

Also, I found a bug in the edge case of doing Response.Redirect() from AJAX (UpdatePanel) - this strips the encoding header but not the filter - hence you get an error at client.

In my case I "solved" this by setting the filter etc in page PreRender rather than Load (Redirect() will hopefully occur before PreRender). I haven't tried the PreSendRequestHeaders method but that may also fix it.

Another handy tip is you can use the DotNetZip library GZipStream which does better compression than the built-in .Net class. http://dotnetzip.codeplex.com/

Cheers
# re: ASP.NET GZip Encoding Caveats
by Mike McGranahan September 02, 2012 @ 11:20pm
Thanks for the write up. Just want to note that the response Vary header should be "Accept-Encoding", not "Content-Encoding". Names listed in the Vary value refer to headers in the Request, not the Response.
# re: ASP.NET GZip Encoding Caveats
by Ian November 28, 2012 @ 9:11am
Hi Rick,

I have implemented this code within Application_BeginRequest on two sites.

One site is MVC 3 asp.net 4 framework.

And the other is a legacy site running on Framework 2.

I have installed both sites onto my local windows 7 premium home addition IIS 7.

The compression works for the MVC 3 site but is completely ignored on the asp.net 2 site.
Looking at Fiddler it seems to swallow the response headers that were added and not compressing anything.

I noticed the dynamic compression module is not installed on my local IIS, does that need to be installed for the compression to work. Even though it works fine for a Framework 4 site.?

Thanks
Ian
# re: ASP.NET GZip Encoding Caveats
by Rick Strahl November 28, 2012 @ 1:35pm
@Ian - dynamic compression does not need to be installed on the server. Dynamic compression is IIS automatically compressing even dynamic content - this code does it 'manually' as part of the individual request processing.

This code should work fine in ASP.NET 2.0, so unless the application is possibly manipulating headers this code should continue to work.
 


West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2014