Contact   •   Articles   •   Products   •   Search

Rick Strahl's Web Log

Wind, waves, code and everything in between...
ASP.NET • C# • HTML5 • JavaScript • AngularJs

HttpWebRequest and GZip Http Responses


I've talked a bit about GZip compression (here and here and here) on the server recently. It's pretty straight forward to use GZip compression either by letting IIS do it for you automatically or by using some simple ASP.NET code to compress content.

As a quick review to create GZip content on the server generically I created a couple simple static utility functions that can be called from anywhere to dynamically encode dynamic ASP.NET content:

/// <summary>
/// Determines if GZip is supported
/// </summary>
/// <returns></returns>
public static bool IsGZipSupported()
{
    string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
    if (!string.IsNullOrEmpty(AcceptEncoding) && 
         ( AcceptEncoding.Contains("gzip") || AcceptEncoding.Contains("deflate") ) )
        return true;
    return false;
}

/// <summary>
/// Sets up the current page or handler to use GZip through a Response.Filter
/// IMPORTANT:  
/// You have to call this method before any output is generated!
/// </summary>
public static void GZipEncodePage()
{
    if (IsGZipSupported())
    {
        HttpResponse Response = HttpContext.Current.Response;                

        string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
        if (AcceptEncoding.Contains("gzip"))
        {
            Response.Filter = new System.IO.Compression.GZipStream(Response.Filter,
                                      System.IO.Compression.CompressionMode.Compress);
            Response.AppendHeader("Content-Encoding", "gzip");
        }
        else
        {
            Response.Filter = new System.IO.Compression.DeflateStream(Response.Filter,
                                      System.IO.Compression.CompressionMode.Compress);
            Response.AppendHeader("Content-Encoding", "deflate");
        }
    }
}

All that needs to happen in an ASP.NET page (or a handler for that matter) is to call GZipEncodePage() before any Response output is written. The single method call  takes care of checking whether the client supports it and then assigning the filter for compression.

The Client Side

This takes care of the server side and if you're using a browser, GZip support is pretty much automatic - it just works as most browsers can decompress Gzip content. But if you're using an HTTP client in the .NET framework - WebClient or HttpWebRequest - you'll find that Gzip is not natively supported. So how do you make your client request GZip content?

You start by adding an Accept-Encoding header to the request that tells the server to encode responses with GZip. That what the server code above is checking for to decide whether GZip needs to be applied. The client then receives the compressed response from the server and needs to decomress it.

Here's what a simplified method for retrieving a URL to string with GZip compression looks like:

/// <summary>
/// Simple routine to retrieve HTTP Content as a string with
/// optional POST data and GZip encoding.
/// </summary>
/// <param name="Url"></param>
/// <param name="PostData"></param>
/// <param name="GZip"></param>
/// <returns></returns>
public string GetUrl(string Url, string PostData, bool GZip)
{
    HttpWebRequest Http = (HttpWebRequest)WebRequest.Create(Url);

    if (GZip)
        Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");

    if (!string.IsNullOrEmpty(PostData))
    {
        Http.Method = "POST";
        byte[] lbPostBuffer = Encoding.Default.GetBytes(PostData);

        Http.ContentLength = lbPostBuffer.Length;

        Stream PostStream = Http.GetRequestStream();
        PostStream.Write(lbPostBuffer, 0, lbPostBuffer.Length);
        PostStream.Close();
    }

    HttpWebResponse WebResponse = (HttpWebResponse)Http.GetResponse();

    Stream responseStream = responseStream = WebResponse.GetResponseStream();
    if (WebResponse.ContentEncoding.ToLower().Contains("gzip"))
        responseStream = new GZipStream(responseStream, CompressionMode.Decompress);
    else if (WebResponse.ContentEncoding.ToLower().Contains("deflate"))
        responseStream = new DeflateStream(responseStream, CompressionMode.Decompress);            

    StreamReader Reader = new StreamReader(responseStream, Encoding.Default);

    string Html = Reader.ReadToEnd();

    WebResponse.Close();
    responseStream.Close();

    return Html;
}

The beauty of the compression streams is that they work on existing streams and so effectively act as a filter for the base stream. This means the compression can usually be plugged into any Stream interface with no other code changes.

It'd be nice if GZip was natively supported by HttpWebRequest, but I suppose this code is easy enough to create a wrapper with. Note that you have to use HttpWebRequest to use GZip encoding - the WebClient class doesn't expose its underlying stream so you can't easily decompress content through that interface.

Make Donation
Posted in .NET  ASP.NET  HTTP  IIS  

The Voices of Reason


 

# DotNetSlackers: HttpWebRequest and GZip Http Responses


Alex O
June 29, 2007

# re: HttpWebRequest and GZip Http Responses

Just an fyi. Gzip is relatively slow as implemented in .NET framework, Deflate is much faster. Plus other implementions of Gzip provide better compression ratios than .NET versions because of patents and such.

Rick Strahl
June 29, 2007

# re: HttpWebRequest and GZip Http Responses

Good point Alex - meant to mention that but forgot. I've actually been using XCeed's compression routines in much of my code, but it's not THAT critical especially in client applications. I suppose you can control the ordering by doing deflate first instead of GZip in the code I posted above as well.

Of course the best thing is to cache content that's been GZipped if possible <s>...

Gopinath V
July 06, 2007

# re: HttpWebRequest and GZip Http Responses

Rick,

Have u tried this under AJAX or update panel senario ? I'm getting

Sys.WebForms.PageRequestManagerParserErrorException: The message received from the server could not be parsed. Common causes for this error are when the response is modified by calls to Response.Write(), response filters, HttpModules, or server trace is enabled.

Thx,

Gopi

Rick Strahl
July 13, 2007

# re: HttpWebRequest and GZip Http Responses

I'm not sure why that should be. If XHR doesn't support GZip encoding (and I think it does) it shouldn't be sending the GZIP accept headers in which case the code above will not encode the response. It's crucial to check whether the browser supports GZip response data and you need to be really careful about caching data in these scenarios as well!

Rick Strahl's Web Log
July 16, 2007

# Rick Strahl's Web Log


Nathaniel
August 13, 2007

# re: HttpWebRequest and GZip Http Responses

I really enjoy your blog and find alot of useful information in it.

I tried your example using it to gzip an xml file that is served by a handler. The resulting output is missing the final close bracket. The document is well formed and the close tag should be there. I ran the same code without letting it compress and the xml file was valid. It seems like the filter is not flushing all the data to the response output stream.

Rick Strahl
August 13, 2007

# re: HttpWebRequest and GZip Http Responses

@Nathaniel - that's odd. are you sure there's nothing truncating your XML string first?

I use this code here for RSS and a ton of other things on this site and I haven't run into any truncated output yet, so I'd be really surprised if there's a flushing problem.

Nathaniel
August 13, 2007

# re: HttpWebRequest and GZip Http Responses

I spent some time and tracked down the problem I was having. It was a flushing problem, of sorts. My handler was calling the Flush() method on the Response object. Calling the Response.Flush() method once you have a compression filter attached will corrupt your output stream.

Rick Strahl
August 13, 2007

# re: HttpWebRequest and GZip Http Responses

Oh, that sounds like a bug in ASP.NET. Flush shouldn't have a negative effect on the filter - have to look at that. Thanks for reporting back that's good to know and have on record here...

Thomas Wetterer
August 30, 2007

# re: HttpWebRequest and GZip Http Responses

Hello,

I use your rountine since you posted it. Now the first request with an not gzip enabled browser happens.
With that I get an error in the IsGZipSupported routine because the AcceptEncoding Object is null. And so AcceptEncoding.Contains does not work.

The request was done with the following code (shortend)
WebClient objWebClient = new WebClient();
aRequestHTML = objWebClient.DownloadData(URL);

With kind regards and greetings from Germany

Thomas

Justin
February 15, 2008

# re: HttpWebRequest and GZip Http Responses

There is a known issue with GZipStream/DeflateStream for truncating response streams when Response.Flush() and Response.End() are used.

See http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=236947

This obviously is going to cause issues with async AJAX requests since the truncated response won't be able to get properly parsed by the script

Srikanth
March 05, 2008

# re: HttpWebRequest and GZip Http Responses

Justion..
FYI
Looks like someone have answered the issue, for Response.Flush() and Response.End()
http://forums.asp.net/p/1043578/1467161.aspx#1467161

daniel
October 03, 2008

# re: HttpWebRequest and GZip Http Responses

fixing bug
sending response as file don't use flush if you are with gzip filter active:



if (!(Response.Filter is System.IO.Compression.GZipStream))
Response.Flush();
Response.End()


I test with .NET2

anhtuan
October 16, 2008

# re: HttpWebRequest and GZip Http Responses

Thank you very much, I'm using your codes for my program.

RB
October 20, 2008

# re: HttpWebRequest and GZip Http Responses

I am using XML Reader to read RSS FEED but it fails if the content is returned gzip encoded. Can any one tell me how to decode or decompress the gzip content before i display it on page. Any sample code to decompress/decode based on the below code.
Sample Code:

try
{
XmlDocument _Doc = new XmlDocument();
string URL = "http://www.afraccess.com/trackrssrssid=21559B02DB32C706EC095C8A55A168EE";

XmlTextReader rssReader = new XmlTextReader(URL);
_Doc.Load(rssReader); // This line fails if the URL returns zgip content
XmlNode nodeChannelTitle = _Doc.SelectSingleNode("rss/channel/title");
if (nodeChannelTitle != null)
{
string Title = nodeChannelTitle.InnerXml;
Response.Write(Title);
}
}
catch
{
Response.Write("Gzip content is not decoded.");
}

Poul Bak
October 26, 2008

# This page is NOT served as gzip (or deflate) !

Am I right if I conclude, that there's too many problems with gzip/deflate compression in asp.net to make it worth while using this code?
So far, I have had only partly success with gzip and ONLY for the html part (no css, js and certainly not dynamic resources). If all other resources must be served without compression, I don't think it's worth the trouble. I mean, if it actually worked, we would all use it. The fact that so few asp.net sites use it, must mean something.

Jader Dias
March 24, 2009

# re: HttpWebRequest and GZip Http Responses

I have disassembled the HttpWebResponse class and it seem do decompress the output automatically:

if (decompressionMethod != DecompressionMethods.None)
{
string str2 = this.m_HttpResponseHeaders["Content-Encoding"];
if (str2 != null)
{
if (((decompressionMethod & DecompressionMethods.GZip) != DecompressionMethods.None) && (str2.IndexOf("gzip") != -1))
{
this.m_ConnectStream = new GZipWrapperStream(this.m_ConnectStream, CompressionMode.Decompress);
this.m_ContentLength = -1L;
this.m_HttpResponseHeaders["Content-Encoding"] = null;
}
else if (((decompressionMethod & DecompressionMethods.Deflate) != DecompressionMethods.None) && (str2.IndexOf("deflate") != -1))
{
this.m_ConnectStream = new DeflateWrapperStream(this.m_ConnectStream, CompressionMode.Decompress);
this.m_ContentLength = -1L;
this.m_HttpResponseHeaders["Content-Encoding"] = null;
}
}
}

Luc
January 21, 2010

# re: HttpWebRequest and GZip Http Responses

Wow this code is awesome. I am working on a finance application which needs very fast access to content available on the internet and using gzip means I can scrape at incredibly fast speeds.

Fabio Zilberman
February 04, 2010

# re: HttpWebRequest and GZip Http Responses

I have an aspx page used to get data from a client program o a PDA.
Now I am posting large strings and wanted to gzip them.
I am able to gzip the request on the PDA but not to un-gzip the full request on the aspx page.

if my code if like this:

<%@ Page Language="VB" validateRequest="false" %>
<%@ Import Namespace="System.Data" %>
<%@ Import Namespace="System.Data.SqlClient" %>
<%@ Import Namespace="System.IO" %>
<%@ Import Namespace="System.IO.Compression" %>

<script language="VB" runat="server">
</script>

<%

' un-gziping code ?

DIM dataA as String = Request.Form("dataA")
DIM dataB as String = Request.Form("dataB")

do something with dataA and dataB

Response.Write("Ok")

%>

how can I un-gzip the Request Stream?

I tried doing this
Request.Filter = New System.IO.Compression.GZipStream(Request.InputStream, System.IO.Compression.CompressionMode.Decompress)

and this
Request.Filter = New System.IO.Compression.GZipStream(Request.Filter, System.IO.Compression.CompressionMode.Decompress)

previous to using the Request.Form() lines, but didn't work

Thanks a lot
Fabio.

asp.net developer
March 01, 2010

# re: HttpWebRequest and GZip Http Responses

Hi,

Works great in IIS6, but it only seems to work inh IIS 7 under classic pipeline mode. Any ideas?

nodmonkey
April 09, 2010

# re: HttpWebRequest and GZip Http Responses

I think your test for client accepting gzip is not standards compliant.

In GZipEncodePage, your test for gzip is to examine the Accept-Encoding header and see if it contains "gzip".

According to RFIC 2616 http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3

1. Accept-Encoding header could contain "gzip;q=0" which would mean that gzip is NOT accepted.

2. It could contain "*;q=1.0", which would mean gzip IS accepted.

3. There may be no header, in which case the standard is to accept any encoding.

Your test would produce incorrect results in each of these three cases.

varadhg
May 14, 2010

# re: HttpWebRequest and GZip Http Responses

Is their a way to set the Compression level in these methods?

Thx

varadhg

Lakshmish
September 20, 2011

# re: HttpWebRequest and GZip Http Responses

I have the following captured using fiddler; the request header - Accept-Encoding: gzip,deflate,sdch and in the Response Header - Content-Encoding: gzip

However following is not working

if (WebResponse.ContentEncoding.ToLower().Contains("gzip"))
responseStream = new GZipStream(responseStream, CompressionMode.Decompress);
else if (WebResponse.ContentEncoding.ToLower().Contains("deflate"))
responseStream = new DeflateStream(responseStream, CompressionMode.Decompress);

It doesn't get thru if condition. If i use GZipStream or DeflateStream to decompress directly, it is going to exception while reading at the end.

Any help is highly appreciated

vipul
May 18, 2012

# re: HttpWebRequest and GZip Http Responses


Hi Rick
I just convert your cod in vb.net and use for gzip http response and request. but my case it is throwing Error " The remote server returned an error: (500) Internal Server Error."

Please Help me. In xml http request response it is getting 150 to 200 Second.


Protected Function PostXml(ByVal url As String, ByVal xml As String) As String
Dim strResult As String = String.Empty

Try
Dim Http As HttpWebRequest = WebRequest.Create(url)

Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")

If Not String.IsNullOrEmpty(xml) Then
Http.Method = "POST"
Dim lbPostBuffer As Byte() = Encoding.Default.GetBytes(xml)

Http.ContentLength = lbPostBuffer.Length

Using PostStream As Stream = Http.GetRequestStream()
PostStream.Write(lbPostBuffer, 0, lbPostBuffer.Length)
End Using
End If

Using WebResponse As HttpWebResponse = Http.GetResponse()

Dim responseStream As Stream = WebResponse.GetResponseStream()

If (WebResponse.ContentEncoding.ToLower().Contains("gzip")) Then
responseStream = New GZipStream(responseStream, CompressionMode.Decompress)
ElseIf (WebResponse.ContentEncoding.ToLower().Contains("deflate")) Then
responseStream = New DeflateStream(responseStream, CompressionMode.Decompress)
End If

Dim reader As StreamReader = New StreamReader(responseStream, Encoding.Default)

Dim html As String = reader.ReadToEnd()

responseStream.Close()

Return html
End Using

Catch ex As WebException
strResult = ""
End Try
Return strResult
End Function

Shiras
July 18, 2013

# re: HttpWebRequest and GZip Http Responses

Hi,

I went through your post. its awesome.

I have a different requirement. I have JQuery AJAX request invoked from HTML pages. in response I get HTML and Javascript in JSON format.

If I have to GZIP this manually ignoring dynamic/static suppression of IIS. How should I do this.

I tried using GZipStream but no luck.

My aim: is to save the GZipped data to the database. so each time I can serve the request from my database instead of executing my business logic again and again.

Please advice.

Thanks,
Shiras

Rick Strahl
July 18, 2013

# re: HttpWebRequest and GZip Http Responses

@Shiras - Not sure I understand your question. Are you trying to GZip the server's response or capture the GZip result on the client to capture or both? 

In general I recommend not using dynamic content compression in IIS because it puts too much stress on the server IMHO as every little page will be compressed. Instead I'd recommend manually picking and choosing the few things that really require compression on the server and caching it if possible.

Static compression on the other should always be enabled as the server pre-encodes and caches that content effectively on disk and serves it just like static files.

HttpWebRequest on the client can easily capture the raw undecoded HTTP content for storage in a database (or whatever binary storage stream). You just need to make sure you don't decode the data first (as this article describes) but instead capture the raw data.

Philip Boone
July 24, 2013

# re: HttpWebRequest and GZip Http Responses

absolutly awesome! Thanks for the code!
 


West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2015