Contact   •   Products   •   Search

Rick Strahl's Web Log

Wind, waves, code and everything in between...
ASP.NET • C# • HTML5 • JavaScript • AngularJs

HttpWebRequest and GZip Http Responses


I've talked a bit about GZip compression (here and here and here) on the server recently. It's pretty straight forward to use GZip compression either by letting IIS do it for you automatically or by using some simple ASP.NET code to compress content.

As a quick review to create GZip content on the server generically I created a couple simple static utility functions that can be called from anywhere to dynamically encode dynamic ASP.NET content:

/// <summary>
/// Determines if GZip is supported
/// </summary>
/// <returns></returns>
public static bool IsGZipSupported()
{
    string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
    if (!string.IsNullOrEmpty(AcceptEncoding) && 
         ( AcceptEncoding.Contains("gzip") || AcceptEncoding.Contains("deflate") ) )
        return true;
    return false;
}

/// <summary>
/// Sets up the current page or handler to use GZip through a Response.Filter
/// IMPORTANT:  
/// You have to call this method before any output is generated!
/// </summary>
public static void GZipEncodePage()
{
    if (IsGZipSupported())
    {
        HttpResponse Response = HttpContext.Current.Response;                

        string AcceptEncoding = HttpContext.Current.Request.Headers["Accept-Encoding"];
        if (AcceptEncoding.Contains("gzip"))
        {
            Response.Filter = new System.IO.Compression.GZipStream(Response.Filter,
                                      System.IO.Compression.CompressionMode.Compress);
            Response.AppendHeader("Content-Encoding", "gzip");
        }
        else
        {
            Response.Filter = new System.IO.Compression.DeflateStream(Response.Filter,
                                      System.IO.Compression.CompressionMode.Compress);
            Response.AppendHeader("Content-Encoding", "deflate");
        }
    }
}

All that needs to happen in an ASP.NET page (or a handler for that matter) is to call GZipEncodePage() before any Response output is written. The single method call  takes care of checking whether the client supports it and then assigning the filter for compression.

The Client Side

This takes care of the server side and if you're using a browser, GZip support is pretty much automatic - it just works as most browsers can decompress Gzip content. But if you're using an HTTP client in the .NET framework - WebClient or HttpWebRequest - you'll find that Gzip is not natively supported. So how do you make your client request GZip content?

You start by adding an Accept-Encoding header to the request that tells the server to encode responses with GZip. That what the server code above is checking for to decide whether GZip needs to be applied. The client then receives the compressed response from the server and needs to decomress it.

Here's what a simplified method for retrieving a URL to string with GZip compression looks like:

/// <summary>
/// Simple routine to retrieve HTTP Content as a string with
/// optional POST data and GZip encoding.
/// </summary>
/// <param name="Url"></param>
/// <param name="PostData"></param>
/// <param name="GZip"></param>
/// <returns></returns>
public string GetUrl(string Url, string PostData, bool GZip)
{
    HttpWebRequest Http = (HttpWebRequest)WebRequest.Create(Url);

    if (GZip)
        Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");

    if (!string.IsNullOrEmpty(PostData))
    {
        Http.Method = "POST";
        byte[] lbPostBuffer = Encoding.Default.GetBytes(PostData);

        Http.ContentLength = lbPostBuffer.Length;

        Stream PostStream = Http.GetRequestStream();
        PostStream.Write(lbPostBuffer, 0, lbPostBuffer.Length);
        PostStream.Close();
    }

    HttpWebResponse WebResponse = (HttpWebResponse)Http.GetResponse();

    Stream responseStream = responseStream = WebResponse.GetResponseStream();
    if (WebResponse.ContentEncoding.ToLower().Contains("gzip"))
        responseStream = new GZipStream(responseStream, CompressionMode.Decompress);
    else if (WebResponse.ContentEncoding.ToLower().Contains("deflate"))
        responseStream = new DeflateStream(responseStream, CompressionMode.Decompress);            

    StreamReader Reader = new StreamReader(responseStream, Encoding.Default);

    string Html = Reader.ReadToEnd();

    WebResponse.Close();
    responseStream.Close();

    return Html;
}

The beauty of the compression streams is that they work on existing streams and so effectively act as a filter for the base stream. This means the compression can usually be plugged into any Stream interface with no other code changes.

It'd be nice if GZip was natively supported by HttpWebRequest, but I suppose this code is easy enough to create a wrapper with. Note that you have to use HttpWebRequest to use GZip encoding - the WebClient class doesn't expose its underlying stream so you can't easily decompress content through that interface.

Make Donation
Posted in .NET  ASP.NET  HTTP  IIS  


Feedback for this Post

 
# DotNetSlackers: HttpWebRequest and GZip Http Responses
by DotNetSlackers Latest ASP.NET News June 29, 2007 @ 4:57am
# re: HttpWebRequest and GZip Http Responses
by Alex O June 29, 2007 @ 7:03am
Just an fyi. Gzip is relatively slow as implemented in .NET framework, Deflate is much faster. Plus other implementions of Gzip provide better compression ratios than .NET versions because of patents and such.
# re: HttpWebRequest and GZip Http Responses
by Rick Strahl June 29, 2007 @ 10:57am
Good point Alex - meant to mention that but forgot. I've actually been using XCeed's compression routines in much of my code, but it's not THAT critical especially in client applications. I suppose you can control the ordering by doing deflate first instead of GZip in the code I posted above as well.

Of course the best thing is to cache content that's been GZipped if possible <s>...
# re: HttpWebRequest and GZip Http Responses
by Gopinath V July 06, 2007 @ 8:37pm
Rick,

Have u tried this under AJAX or update panel senario ? I'm getting

Sys.WebForms.PageRequestManagerParserErrorException: The message received from the server could not be parsed. Common causes for this error are when the response is modified by calls to Response.Write(), response filters, HttpModules, or server trace is enabled.

Thx,

Gopi
# re: HttpWebRequest and GZip Http Responses
by Rick Strahl July 13, 2007 @ 1:55am
I'm not sure why that should be. If XHR doesn't support GZip encoding (and I think it does) it shouldn't be sending the GZIP accept headers in which case the code above will not encode the response. It's crucial to check whether the browser supports GZip response data and you need to be really careful about caching data in these scenarios as well!
# Rick Strahl's Web Log
by Rick Strahl's Web Log July 16, 2007 @ 5:32am
# re: HttpWebRequest and GZip Http Responses
by Nathaniel August 13, 2007 @ 1:42pm
I really enjoy your blog and find alot of useful information in it.

I tried your example using it to gzip an xml file that is served by a handler. The resulting output is missing the final close bracket. The document is well formed and the close tag should be there. I ran the same code without letting it compress and the xml file was valid. It seems like the filter is not flushing all the data to the response output stream.
# re: HttpWebRequest and GZip Http Responses
by Rick Strahl August 13, 2007 @ 1:51pm
@Nathaniel - that's odd. are you sure there's nothing truncating your XML string first?

I use this code here for RSS and a ton of other things on this site and I haven't run into any truncated output yet, so I'd be really surprised if there's a flushing problem.
# re: HttpWebRequest and GZip Http Responses
by Nathaniel August 13, 2007 @ 4:13pm
I spent some time and tracked down the problem I was having. It was a flushing problem, of sorts. My handler was calling the Flush() method on the Response object. Calling the Response.Flush() method once you have a compression filter attached will corrupt your output stream.
# re: HttpWebRequest and GZip Http Responses
by Rick Strahl August 13, 2007 @ 4:33pm
Oh, that sounds like a bug in ASP.NET. Flush shouldn't have a negative effect on the filter - have to look at that. Thanks for reporting back that's good to know and have on record here...
# re: HttpWebRequest and GZip Http Responses
by Thomas Wetterer August 30, 2007 @ 12:18am
Hello,

I use your rountine since you posted it. Now the first request with an not gzip enabled browser happens.
With that I get an error in the IsGZipSupported routine because the AcceptEncoding Object is null. And so AcceptEncoding.Contains does not work.

The request was done with the following code (shortend)
WebClient objWebClient = new WebClient();
aRequestHTML = objWebClient.DownloadData(URL);

With kind regards and greetings from Germany

Thomas
# re: HttpWebRequest and GZip Http Responses
by Justin February 15, 2008 @ 1:04pm
There is a known issue with GZipStream/DeflateStream for truncating response streams when Response.Flush() and Response.End() are used.

See http://connect.microsoft.com/VisualStudio/feedback/ViewFeedback.aspx?FeedbackID=236947

This obviously is going to cause issues with async AJAX requests since the truncated response won't be able to get properly parsed by the script
# re: HttpWebRequest and GZip Http Responses
by Srikanth March 05, 2008 @ 4:37am
Justion..
FYI
Looks like someone have answered the issue, for Response.Flush() and Response.End()
http://forums.asp.net/p/1043578/1467161.aspx#1467161
# re: HttpWebRequest and GZip Http Responses
by daniel October 03, 2008 @ 2:04pm
fixing bug
sending response as file don't use flush if you are with gzip filter active:



if (!(Response.Filter is System.IO.Compression.GZipStream))
Response.Flush();
Response.End()


I test with .NET2
# re: HttpWebRequest and GZip Http Responses
by anhtuan October 16, 2008 @ 3:53am
Thank you very much, I'm using your codes for my program.
# re: HttpWebRequest and GZip Http Responses
by RB October 20, 2008 @ 11:03am
I am using XML Reader to read RSS FEED but it fails if the content is returned gzip encoded. Can any one tell me how to decode or decompress the gzip content before i display it on page. Any sample code to decompress/decode based on the below code.
Sample Code:

try
{
XmlDocument _Doc = new XmlDocument();
string URL = "http://www.afraccess.com/trackrssrssid=21559B02DB32C706EC095C8A55A168EE";

XmlTextReader rssReader = new XmlTextReader(URL);
_Doc.Load(rssReader); // This line fails if the URL returns zgip content
XmlNode nodeChannelTitle = _Doc.SelectSingleNode("rss/channel/title");
if (nodeChannelTitle != null)
{
string Title = nodeChannelTitle.InnerXml;
Response.Write(Title);
}
}
catch
{
Response.Write("Gzip content is not decoded.");
}
# This page is NOT served as gzip (or deflate) !
by Poul Bak October 26, 2008 @ 3:48am
Am I right if I conclude, that there's too many problems with gzip/deflate compression in asp.net to make it worth while using this code?
So far, I have had only partly success with gzip and ONLY for the html part (no css, js and certainly not dynamic resources). If all other resources must be served without compression, I don't think it's worth the trouble. I mean, if it actually worked, we would all use it. The fact that so few asp.net sites use it, must mean something.
# re: HttpWebRequest and GZip Http Responses
by Jader Dias March 24, 2009 @ 11:25am
I have disassembled the HttpWebResponse class and it seem do decompress the output automatically:

if (decompressionMethod != DecompressionMethods.None)
{
string str2 = this.m_HttpResponseHeaders["Content-Encoding"];
if (str2 != null)
{
if (((decompressionMethod & DecompressionMethods.GZip) != DecompressionMethods.None) && (str2.IndexOf("gzip") != -1))
{
this.m_ConnectStream = new GZipWrapperStream(this.m_ConnectStream, CompressionMode.Decompress);
this.m_ContentLength = -1L;
this.m_HttpResponseHeaders["Content-Encoding"] = null;
}
else if (((decompressionMethod & DecompressionMethods.Deflate) != DecompressionMethods.None) && (str2.IndexOf("deflate") != -1))
{
this.m_ConnectStream = new DeflateWrapperStream(this.m_ConnectStream, CompressionMode.Decompress);
this.m_ContentLength = -1L;
this.m_HttpResponseHeaders["Content-Encoding"] = null;
}
}
}
# re: HttpWebRequest and GZip Http Responses
by Luc January 21, 2010 @ 8:20pm
Wow this code is awesome. I am working on a finance application which needs very fast access to content available on the internet and using gzip means I can scrape at incredibly fast speeds.
# re: HttpWebRequest and GZip Http Responses
by Fabio Zilberman February 04, 2010 @ 6:55am
I have an aspx page used to get data from a client program o a PDA.
Now I am posting large strings and wanted to gzip them.
I am able to gzip the request on the PDA but not to un-gzip the full request on the aspx page.

if my code if like this:

<%@ Page Language="VB" validateRequest="false" %>
<%@ Import Namespace="System.Data" %>
<%@ Import Namespace="System.Data.SqlClient" %>
<%@ Import Namespace="System.IO" %>
<%@ Import Namespace="System.IO.Compression" %>

<script language="VB" runat="server">
</script>

<%

' un-gziping code ?

DIM dataA as String = Request.Form("dataA")
DIM dataB as String = Request.Form("dataB")

do something with dataA and dataB

Response.Write("Ok")

%>

how can I un-gzip the Request Stream?

I tried doing this
Request.Filter = New System.IO.Compression.GZipStream(Request.InputStream, System.IO.Compression.CompressionMode.Decompress)

and this
Request.Filter = New System.IO.Compression.GZipStream(Request.Filter, System.IO.Compression.CompressionMode.Decompress)

previous to using the Request.Form() lines, but didn't work

Thanks a lot
Fabio.
# re: HttpWebRequest and GZip Http Responses
by asp.net developer March 01, 2010 @ 3:05am
Hi,

Works great in IIS6, but it only seems to work inh IIS 7 under classic pipeline mode. Any ideas?
# re: HttpWebRequest and GZip Http Responses
by nodmonkey April 09, 2010 @ 5:09am
I think your test for client accepting gzip is not standards compliant.

In GZipEncodePage, your test for gzip is to examine the Accept-Encoding header and see if it contains "gzip".

According to RFIC 2616 http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3

1. Accept-Encoding header could contain "gzip;q=0" which would mean that gzip is NOT accepted.

2. It could contain "*;q=1.0", which would mean gzip IS accepted.

3. There may be no header, in which case the standard is to accept any encoding.

Your test would produce incorrect results in each of these three cases.
# re: HttpWebRequest and GZip Http Responses
by varadhg May 14, 2010 @ 6:34pm
Is their a way to set the Compression level in these methods?

Thx

varadhg
# re: HttpWebRequest and GZip Http Responses
by Lakshmish September 20, 2011 @ 8:55am
I have the following captured using fiddler; the request header - Accept-Encoding: gzip,deflate,sdch and in the Response Header - Content-Encoding: gzip

However following is not working

if (WebResponse.ContentEncoding.ToLower().Contains("gzip"))
responseStream = new GZipStream(responseStream, CompressionMode.Decompress);
else if (WebResponse.ContentEncoding.ToLower().Contains("deflate"))
responseStream = new DeflateStream(responseStream, CompressionMode.Decompress);

It doesn't get thru if condition. If i use GZipStream or DeflateStream to decompress directly, it is going to exception while reading at the end.

Any help is highly appreciated
# re: HttpWebRequest and GZip Http Responses
by vipul May 18, 2012 @ 3:10am

Hi Rick
I just convert your cod in vb.net and use for gzip http response and request. but my case it is throwing Error " The remote server returned an error: (500) Internal Server Error."

Please Help me. In xml http request response it is getting 150 to 200 Second.


Protected Function PostXml(ByVal url As String, ByVal xml As String) As String
Dim strResult As String = String.Empty

Try
Dim Http As HttpWebRequest = WebRequest.Create(url)

Http.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate")

If Not String.IsNullOrEmpty(xml) Then
Http.Method = "POST"
Dim lbPostBuffer As Byte() = Encoding.Default.GetBytes(xml)

Http.ContentLength = lbPostBuffer.Length

Using PostStream As Stream = Http.GetRequestStream()
PostStream.Write(lbPostBuffer, 0, lbPostBuffer.Length)
End Using
End If

Using WebResponse As HttpWebResponse = Http.GetResponse()

Dim responseStream As Stream = WebResponse.GetResponseStream()

If (WebResponse.ContentEncoding.ToLower().Contains("gzip")) Then
responseStream = New GZipStream(responseStream, CompressionMode.Decompress)
ElseIf (WebResponse.ContentEncoding.ToLower().Contains("deflate")) Then
responseStream = New DeflateStream(responseStream, CompressionMode.Decompress)
End If

Dim reader As StreamReader = New StreamReader(responseStream, Encoding.Default)

Dim html As String = reader.ReadToEnd()

responseStream.Close()

Return html
End Using

Catch ex As WebException
strResult = ""
End Try
Return strResult
End Function
# re: HttpWebRequest and GZip Http Responses
by Shiras July 18, 2013 @ 7:26am
Hi,

I went through your post. its awesome.

I have a different requirement. I have JQuery AJAX request invoked from HTML pages. in response I get HTML and Javascript in JSON format.

If I have to GZIP this manually ignoring dynamic/static suppression of IIS. How should I do this.

I tried using GZipStream but no luck.

My aim: is to save the GZipped data to the database. so each time I can serve the request from my database instead of executing my business logic again and again.

Please advice.

Thanks,
Shiras
# re: HttpWebRequest and GZip Http Responses
by Rick Strahl July 18, 2013 @ 8:42am
@Shiras - Not sure I understand your question. Are you trying to GZip the server's response or capture the GZip result on the client to capture or both? In general I recommend not using dynamic content compression in IIS because it puts too much stress on the server IMHO. Instead I'd recommend manually picking and choosing the few things that really require compression on the server and caching it if possible. Static compression on the other should be enabled as the server pre-encodes and caches that content effectively.

HttpWebRequest on the client can easily capture the raw undecoded HTTP content for storage in a database (or whatever binary storage stream). You just need to make sure you don't decode the data first (as this article describes) but instead capture the raw data.
# re: HttpWebRequest and GZip Http Responses
by Philip Boone July 24, 2013 @ 8:34am
absolutly awesome! Thanks for the code!
 


West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2014