Rick Strahl's Weblog  

Wind, waves, code and everything in between...
.NET • C# • Markdown • WPF • All Things Web
Contact   •   Articles   •   Products   •   Support   •   Advertise
Sponsored by:
Markdown Monster - The Markdown Editor for Windows

ASP.NET GZip Encoding Caveats


:P
On this page:

GZip encoding in ASP.NET is pretty easy to accomplish using the built-in GZipStream and DeflateStream classes and applying them to the Response.Filter property.  While applying GZip and Deflate behavior is pretty easy there are a few caveats that you have watch out for as I found out today for myself with an application that was throwing up some garbage data. But before looking at caveats let’s review GZip implementation for ASP.NET.

ASP.NET GZip/Deflate Basics

Response filters basically are applied to the Response.OutputStream and transform it as data is written to it through the ASP.NET Response object. So a Response.Write eventually gets written into the output stream which if a filter is also written through the filter stream’s interface. To perform the actual GZip (and Deflate) encoding typically used by Web pages .NET includes the GZipStream and DeflateStream stream classes which can be readily assigned to the Repsonse.OutputStream.

With these two stream classes in place it’s almost trivially easy to create a couple of reusable methods that allow you to compress your HTTP output. In my standard WebUtils utility class (from the West Wind West Wind Web Toolkit) created two static utility methods – IsGZipSupported and GZipEncodePage – that check whether the client supports GZip encoding and then actually encodes the current output (note that although the method includes ‘Page’ in its name this code will work with any ASP.NET output).

/// <summary>
/// Determines if GZip is supported
/// </summary>
/// <returns></returns>
public static bool IsGZipSupported()
{
    string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
    if (!string.IsNullOrEmpty(AcceptEncoding) &&
            (AcceptEncoding.Contains("gzip") || AcceptEncoding.Contains("deflate")))
        return true;
    return false;
}

/// <summary>
/// Sets up the current page or handler to use GZip through a Response.Filter
/// IMPORTANT:  
/// You have to call this method before any output is generated!
/// </summary>
public static void GZipEncodePage()
{
    HttpResponse Response = HttpContext.Current.Response;

    if (IsGZipSupported())
    {
        string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];

        if (AcceptEncoding.Contains("deflate"))
        {
            Response.Filter = new System.IO.Compression.DeflateStream(Response.Filter,
                                        System.IO.Compression.CompressionMode.Compress);
            Response.Headers.Remove("Content-Encoding");
            Response.AppendHeader("Content-Encoding", "deflate");
        }
        else
        {
            Response.Filter = new System.IO.Compression.GZipStream(Response.Filter,
                                        System.IO.Compression.CompressionMode.Compress);
            Response.Headers.Remove("Content-Encoding");
            Response.AppendHeader("Content-Encoding", "gzip");
        }

    }
}

As you can see the actual assignment of the Filter is as simple as:

Response.Filter = new DeflateStream(Response.Filter, System.IO.Compression.CompressionMode.Compress);

which applies the filter to the OutputStream. You also need to ensure that your response reflects the new GZip or Deflate encoding and ensure that any pages that are cached in Proxy servers can differentiate between pages that were encoded with the various different encodings (or no encoding).

To use this utility function now is trivially easy: In any ASP.NET code that wants to compress its Response output you simply use:

protected void Page_Load(object sender, EventArgs e)
{            
    WebUtils.GZipEncodePage();

    Entry = WebLogFactory.GetEntry();

    var entries = Entry.GetLastEntries(App.Configuration.ShowEntryCount, "pk,Title,SafeTitle,Body,Entered,Feedback,Location,ShowTopAd", "TEntries");
    if (entries == null)
        throw new ApplicationException("Couldn't load WebLog Entries: " + Entry.ErrorMessage);

    this.repEntries.DataSource = entries;
    this.repEntries.DataBind();

}

Here I use an ASP.NET page, but the above WebUtils.GZipEncode() method call will work in any ASP.NET application type including HTTP Handlers. The only requirement is that the filter needs to be applied before any other output is sent to the OutputStream. For example, in my CallbackHandler service implementation by default output over a certain size is GZip encoded. The output that is generated is JSON or XML and if the output is over 5k in size I apply WebUtils.GZipEncode():

if (sbOutput.Length > GZIP_ENCODE_TRESHOLD)
    WebUtils.GZipEncodePage();

Response.ContentType = ControlResources.STR_JsonContentType;
HttpContext.Current.Response.Write(sbOutput.ToString());

Ok, so you probably get the idea: Encoding GZip/Deflate content is pretty easy.

Hold on there Hoss –Watch your Caching

Or is it? There are a few caveats that you need to watch out for when dealing with GZip content. The fist issue is that you need to deal with the fact that some clients don’t support GZip or Deflate content. Most modern browsers support it, but if you have a programmatic Http client accessing your content GZip/Deflate support is by no means guaranteed. For example, WinInet Http clients don’t support GZip out of the box – it has to be explicitly implemented. Other low level HTTP clients on other platforms too don’t support GZip out of the box.

The problem is that your application, your Web Server and Proxy Servers on the Internet might be caching your generated content. If you return content with GZip once and then again without, either caching is not applied or worse the wrong type of content is returned back to the client from a cache or proxy. The result is an unreadable response for *some clients* which is also very hard to debug and fix once in production.

You already saw the issue of Proxy servers addressed in the GZipEncodePage() function:

// Allow proxy servers to cache encoded and unencoded versions separately
Response.AppendHeader("Vary", "Content-Encoding");

This ensures that any Proxy servers also check for the Content-Encoding HTTP Header to cache their content – not just the URL.

The same thing applies if you do OutputCaching in your own ASP.NET code. If you generate output for GZip on an OutputCached page the GZipped content will be cached (either by ASP.NET’s cache or in some cases by the IIS Kernel Cache). But what if the next client doesn’t support GZip? She’ll get served a cached GZip page that won’t decode and she’ll get a page full of garbage. Wholly undesirable. To fix this you need to add some custom OutputCache rules by way of the GetVaryByCustom() HttpApplication method in your global_ASAX file:

public override string GetVaryByCustomString(HttpContext context, string custom)
{
    // Override Caching for compression
    if (custom == "GZIP")
    {
        string acceptEncoding = HttpContext.Current.Response.Headers["Content-Encoding"];
        
        if (string.IsNullOrEmpty(acceptEncoding))
            return "";
        else if (acceptEncoding.Contains("gzip"))
            return "GZIP";
        else if (acceptEncoding.Contains("deflate"))
            return "DEFLATE";
        
        return "";
    }
    
    return base.GetVaryByCustomString(context, custom);
}

In a page that use Output caching you then specify:

<%@ OutputCache Duration="180" VaryByParam="none" VaryByCustom="GZIP" %>

To use that custom rule.

It’s all Fun and Games until ASP.NET throws an Error

Ok, so you’re up and running with GZip, you have your caching squared away and your pages that you are applying it to are jamming along. Then BOOM, something strange happens and you get a lovely garbled page that look like this:

GarbledOutput

Lovely isn’t it?

What’s happened here is that I have WebUtils.GZipEncode() applied to my page, but there’s an error in the page. The error falls back to the ASP.NET error handler and the error handler removes all existing output (good) and removes all the custom HTTP headers I’ve set manually (usually good, but very bad here). Since I applied the Response.Filter (via GZipEncode) the output is now GZip encoded, but ASP.NET has removed my Content-Encoding header, so the browser receives the GZip encoded content without a notification that it is encoded as GZip. The result is binary output. Here’s what Fiddler says about the raw HTTP header output when an error occurs when GZip encoding was applied:

HTTP/1.1 500 Internal Server Error
Cache-Control: private
Content-Type: text/html; charset=utf-8
Date: Sat, 30 Apr 2011 22:21:08 GMT
Content-Length: 2138
Connection: close

�`I�%&/m�{J�J��t��` … binary output striped here

Notice: no Content-Encoding header and that’s why we’re seeing this garbage. ASP.NET has stripped the Content-Encoding header but left our filter intact.

So how do we fix this? In my applications I typically have a global Application_Error handler set up and in this case I’ve been using that. One thing that you can do in the Application_Error handler is explicitly clear out the Response.Filter and set it to null at the top:

protected void Application_Error(object sender, EventArgs e)
{
        // Remove any special filtering especially GZip filtering
        Response.Filter = null;
}

And voila I get my Yellow Screen of Death or my custom generated error output back via uncompressed content. BTW, the same is true for Page level errors handled in Page_Error or ASP.NET MVC Error handling methods in a controller.

Another and possibly even better solution is to check whether a filter is attached just before the headers are sent to the client as pointed out by Adam Schroeder in the comments:

 protected void Application_PreSendRequestHeaders()
{
    // ensure that if GZip/Deflate Encoding is applied that headers are set
    // also works when error occurs if filters are still active
    HttpResponse response = HttpContext.Current.Response;
    if (response.Filter is GZipStream && response.Headers["Content-encoding"] != "gzip")
        response.AppendHeader("Content-encoding", "gzip");
    else if (response.Filter is DeflateStream && response.Headers["Content-encoding"] != "deflate")
        response.AppendHeader("Content-encoding", "deflate");
}

This uses the Application_PreSendRequestHeaders() pipeline event to check for compression encoding in a filter and adjusts the content accordingly. This is actually a better solution since this is generic – it’ll work regardless of how the content is cleaned up. For example, an error Response.Redirect() or short error display might get changed and the filter not cleared and this code actually handles that. Sweet, thanks Adam.

It’s unfortunate that ASP.NET doesn’t natively clear out Response.Filters when an error occurs just as it clears the Response and Headers. I can’t see where leaving a Filter in place in an error situation would make any sense, but hey - this is what it is and it’s easy enough to fix as long as you know where to look. Riiiight!

IIS and GZip

I should also mention that IIS 7 includes good support for compression natively. If you can defer encoding to let IIS perform it for you rather than doing it in your code by all means you should do it! Especially any static or semi-dynamic content that can be made static should be using IIS built-in compression. Dynamic caching is also supported but is a bit more tricky to judge in terms of performance and footprint. John Forsyth has a great article on the benefits and drawbacks of IIS 7 compression which gives some detailed performance comparisons and impact reviews. I’ll post another entry next with some more info on IIS compression since information on it seems to be a bit hard to come by.

Related Content

Posted in ASP.NET  IIS7  

The Voices of Reason


 

Adam Schroder
May 02, 2011

# re: ASP.NET GZip Encoding Caveats

I was working on this same issue today and solved it like this (opposite of your solution). I think that your solution is probably better. It really is a shame it doesn't do this automatically.


   protected void Application_PreSendRequestHeaders()
    {
        HttpResponse response = HttpContext.Current.Response;
        if (response.Filter is GZipStream && response.Headers["Content-encoding"] != "gzip")
            response.AppendHeader("Content-encoding", "gzip");
        else if (response.Filter is DeflateStream && response.Headers["Content-encoding"] != "deflate")
            response.AppendHeader("Content-encoding", "deflate");
    }
 

Rick Strahl
May 02, 2011

# re: ASP.NET GZip Encoding Caveats

@Adam - actually that's a pretty sweet way to add the headers and that's actually more reliable since it captures all scenarios where the response may be cleared out inadvertantly.

For example I have various service implementation where handlers manage error trapping and error return and there too I know I've forgotten in the past to clear out filters and the above would capture that.

lonely soul
May 06, 2011

# re: ASP.NET GZip Encoding Caveats

great article as usual Rick.

Matt
May 09, 2011

# re: ASP.NET GZip Encoding Caveats

I think you need to detect IE6 clients also and not serve gzipped content to them. My understanding is that IE6 (prior to SP2?) will report that it can accept gzip content but will screw up the decoding of the compressed content under various, seemingly random, scenarios.

Andy
May 09, 2011

# re: ASP.NET GZip Encoding Caveats

Adam's solution is very reliable but with one caveat... The GZipStream must be the last filter in the Response.Filter stream chain. (Though I can't think if any situation where it wouldn't be...) But it's possible that streams are chained after the GZipStream to the Response.Filter, therefore getting the Response.Filter before sending headers will return whichever stream was last applied.

For example consider:
Response.Filter = new System.IO.Compression.GZipStream(Response.Filter,
     System.IO.Compression.CompressionMode.Compress);

... later in the same request execution ...
Response.Filter = new MyFilter2(Response.Filter);

then...
(Response.Filter is GZipStream) == false;

vimal
May 10, 2011

# re: ASP.NET GZip Encoding Caveats

Very nice article

Cbrcoder
May 16, 2011

# re: ASP.NET GZip Encoding Caveats

Don't you think this is a webserver concern and best not to use it in code ? Just make a comparison of performance difference between IIS 7.5 and .NET version of compression libraries, there is atleast 10x difference in speed. It's best for programmers to not mess with what is a SERVER concern.

Rick Strahl
May 17, 2011

# re: ASP.NET GZip Encoding Caveats

@CBrCoder - I don't agree with you. IIS's dynamic compression gives you no control over what gets compressed - it's all or nothing. In my apps where I use compression of dynamic content I really just want a few things compressed not everything. Additionally a lot of the overhead of compression can be negated by properly using Caching in your applications.

FWIW, I posted a second blog entry that talks about built-in compression - choices are good :-)

novi4ok
July 06, 2011

# re: ASP.NET GZip Encoding Caveats

Hello!
I have one question about this code...
I have some errors, because i cant find what i need to write in
using
line...
for example my Visual Studio underline this code
Entry = WebLogFactory.GetEntry();
. I dont know where is WebLogFactory, in which using it is?

Andr
July 12, 2011

# re: ASP.NET GZip Encoding Caveats

Rick, you are very cool guy. You as usual save my hours and even days of dumb work by your great articles.

kralcafe
July 28, 2011

# re: ASP.NET GZip Encoding Caveats

I think you need to detect IE6 clients also and not serve gzipped content to them.

Wynand Murray
September 01, 2011

# re: ASP.NET GZip Encoding Caveats

Hi Rick, thanks for a great post, this has REALLY brought down my site's traffic in size!
I did find one error in your "IsGZipSupported" function though:

I am referring to the following line:

if (!string.IsNullOrEmpty(AcceptEncoding) && (AcceptEncoding.Contains("gzip") || AcceptEncoding.Contains("deflate")))

The exception occurs when the client browser does NOT return an Encoding Header.
Upon checking if the "AcceptEncoding" value (Which will be null) "Contains" some text, an Object Reference exception is thrown.

I fixed this by checking if "AcceptEncoding" is null WITHOUT using the "&&" operator to search for the text value inside it. I only use the "Contains" method to search for values after "AcceptEncoding" has been identified as having a value.

This was strange to me since I didn't think that testing for a "Null" AND a value with the "&&" operator would result in an exception in this case.

Thanks for all your great posts, please keep them coming :)

Regards,

Wynand Murray.

Rick Strahl
September 01, 2011

# re: ASP.NET GZip Encoding Caveats

@Wyand - I'm not sure why that should fail. If the value is null the condition !string.IsNullOrEmpty() will fail and should terminate the if expression via short circuiting - the && should never evaluate. If it indeed fails then there'd be a serious bug in the .NET compiler not properly short circuiting which would break lots of code. You sure there's not something else going on when this fails?

blueray
September 28, 2011

# re: ASP.NET GZip Encoding Caveats

Hi,

excellent article, thanks.

I got problem though. I used the workaround for appending lost content-encoding header by Adam Schroder , but HttpContext.Current is always null inside Application_PreSendRequestHeaders, thus on exception I receive that scrambled text. Any ideas why HttpContext.Current is null ?

DotNetWise
March 24, 2012

# re: ASP.NET GZip Encoding Caveats

Weird enough, ASP.NET MVC 4 Web API ignores IIS gzip compression.
Any ideas how to turn it on in that case too?

Rick Strahl
March 24, 2012

# re: ASP.NET GZip Encoding Caveats

@DotNetWise - what gets ignored exactly? Are you talking about IIS dynamic compression, static compression, ASP.NET based filter GZip compression?

DotNetWise
March 26, 2012

# re: ASP.NET GZip Encoding Caveats

IIS' dynamic compression is ignored on ASP.NET MVC 4 for Actions who return JsonResult or application/json; charset=UTF-8; although the web.config explicitely enables dynamic compression to *.*

<httpCompression sendCacheHeaders="true" directory="c:\temp" minFileSizeForComp="10" noCompressionForProxies="false">
      <clear/>
      <scheme name="gzip" dll="%Windir%\system32\inetsrv\gzip.dll" staticCompressionLevel="9" />
      <dynamicTypes>
        <clear/>
        <add mimeType="text/*" enabled="true" />
        <add mimeType="message/*" enabled="true" />
        <add mimeType="application/x-javascript" enabled="true" />
        <add mimeType="application/json" enabled="true" />
        <add mimeType="application/json; charset=UTF-8" enabled="true"   />
        <add mimeType="*/*" enabled="true" />
      </dynamicTypes>
      <staticTypes>
        <add mimeType="text/*" enabled="true" />
        <add mimeType="message/*" enabled="true" />
        <add mimeType="application/x-javascript" enabled="true" />
        <add mimeType="application/atom+xml" enabled="true" />
        <add mimeType="application/xaml+xml" enabled="true" />
        <add mimeType="application/javascript" enabled="true" />
        <add mimeType="application/javascript; charset=UTF-8" enabled="true" />
        <add mimeType="application/json" enabled="true" />
        <add mimeType="application/json; charset=UTF-8" enabled="true" />
        <add mimeType="*/*" enabled="true" />
      </staticTypes>
    </httpCompression>

James Stoertz
April 12, 2012

# re: ASP.NET GZip Encoding Caveats

Rick, you are brilliant. BRILLIANT!
We had the "garbled" problem (on errors) for months and couldn't resolve it. Couldn't even search for it because of the binary screen.

Lachlan
July 05, 2012

# re: ASP.NET GZip Encoding Caveats

Hi, good article thanks.

I tried throwing an error after setting the filter without doing anything clever, and it seemed to strip the filter itself without any intervention from me?

Also, I found a bug in the edge case of doing Response.Redirect() from AJAX (UpdatePanel) - this strips the encoding header but not the filter - hence you get an error at client.

In my case I "solved" this by setting the filter etc in page PreRender rather than Load (Redirect() will hopefully occur before PreRender). I haven't tried the PreSendRequestHeaders method but that may also fix it.

Another handy tip is you can use the DotNetZip library GZipStream which does better compression than the built-in .Net class. http://dotnetzip.codeplex.com/

Cheers

Mike McGranahan
September 02, 2012

# re: ASP.NET GZip Encoding Caveats

Thanks for the write up. Just want to note that the response Vary header should be "Accept-Encoding", not "Content-Encoding". Names listed in the Vary value refer to headers in the Request, not the Response.

Ian
November 28, 2012

# re: ASP.NET GZip Encoding Caveats

Hi Rick,

I have implemented this code within Application_BeginRequest on two sites.

One site is MVC 3 asp.net 4 framework.

And the other is a legacy site running on Framework 2.

I have installed both sites onto my local windows 7 premium home addition IIS 7.

The compression works for the MVC 3 site but is completely ignored on the asp.net 2 site.
Looking at Fiddler it seems to swallow the response headers that were added and not compressing anything.

I noticed the dynamic compression module is not installed on my local IIS, does that need to be installed for the compression to work. Even though it works fine for a Framework 4 site.?

Thanks
Ian

Rick Strahl
November 28, 2012

# re: ASP.NET GZip Encoding Caveats

@Ian - dynamic compression does not need to be installed on the server. Dynamic compression is IIS automatically compressing even dynamic content - this code does it 'manually' as part of the individual request processing.

This code should work fine in ASP.NET 2.0, so unless the application is possibly manipulating headers this code should continue to work.

Carl-Erik Kopseng
October 10, 2016

# re: ASP.NET GZip Encoding Caveats

Yeah, indeed it is easy, but unfortunately your example code is incorrect, as it does not try to parse the Accept-Encoding per the spec :-/ A http client might say "while this encoding is supported, do not try to use it" by saying "my-format;q=0". Your code fails (gives incorrect results), for instance, on strings like these: "deflate;q=0.0,gzip;q=0.0,identity". Here neither gzip nor deflate should be used.

Dave Transom has some good code for parsing and handling this on his blog where he goes into detail on how to handle this on a more general basis:
http://www.singular.co.nz/2008/07/finding-preferred-accept-encoding-header-in-csharp/

Sivan Leoni
September 06, 2017

# re: ASP.NET GZip Encoding Caveats

Thank you for the great post. Any idea why would a certain website look fine on all computers and browsers, but a specific computer it is garbled text page in all browsers. Is there a settings somewhere in windows that prevents pages from interpreting gzip content correctly? Why would only one of my clients receive the garbled text?

I'm a web programmer, and my application works fine for all people, but I have one client that says that in all their browsers, the page looks garbled.

Any help would be great.

Thanks! Sivan


West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2024