Http monitoring tools are essential when doing any kind of Web development whether it’s for plain Web development, Web Services or any sort of HTTP client work. When things go wrong it is often highly useful to take a look under the hood and dig into the raw HTTP request data to see what HTTP headers were sent from the client to server and what headers and responses come back. 

For plain Web development most of the time I actually use FireBug inside of FireFox, but when more detailed HTTP wire debugging is required I quickly revert to Fiddler, because it tends to provide more information and options on dealing with request data. But even more importantly, FireBug – cool and useful as it is as the Swiss Army knife for Web development debugging - is limited to life inside of FireFox, so it does nada when you need to look at content outside of the browser such as when a .NET application fires HTTP requests against a server.

One of the reasons that Fiddler works so well is because it has a nice, simple and very usable UI that makes it easy to monitor HTTP requests. It also includes a nice request builder which is great for building up requests for testing without having to run code to generate a request against a service for example. In addition there are a few handy viewers that simplify viewing common types of content.

Before I get into the discussion of using Fiddler with .NET HTTP clients like WCF or Web Service proxies or WebClient/HttpWebRequest objects, here are a few basic tips when using Fiddler for debugging in general:

Running Fiddler against local Addresses

If you’re using Fiddler with a localhost or 127.0.0.1 address you’ll find that Fiddler will not monitor requests. The reason for this is that local addresses typically bypass any proxies (and the network adapter in general) and so localhost or 127.0.0.1 doesn’t show up.

Luckily there’s an easy workaround for this: Instead of using a local address use the NetBios Machine name or the local IP address:

http://rasnote/weblog/default.aspx

or

http://192.168.0.104/weblog/default.aspx

And voila you have access to local addresses.

Visual Studio Web Server (Cassini)

What about the ASP.NET local Web Server (Cassini)? There’s a nasty hack for accessing Cassini requests through Fiddler:

http://127.0.0.1.:88/default.aspx

Notice the . after the 127.0.0.1 and before the port number. I’ve only had luck in using 127.0.0.1.  as the domain/address -localhost or the local machine name don’t work.

Using Fiddler with other Browsers

Fiddler works automatically with Internet Explorer. When Fiddler and IE are both open requests made in IE are automatically monitored in Fiddler (unless you explicitly turn monitoring off). You can also use Fiddler with other browsers by configuring the proxy settings manually or by running Fiddlers configuration script.

Fiddler by default uses the proxy address of 127.0.0.1 and port 8888. You can manually configure this in just about any browser. For example in FireFox you can set up the proxy settings like this:

Alternately you can use Fiddler’s startup configuration script which can be fired when FireFox starts. You can set this up on the dialog above like this:

The full script reference looks like this:

file:///C:/Users/rstrahl/Documents/Fiddler2/Scripts/BrowserPAC.js

When set this script is set FireFox checks if Fiddler is running when it starts up and if so sets the Proxy automatically. Otherwise no proxy is set. However, unlike IE this setup will not detect when Fiddler is shut down so requests will try to use the proxy that is then no longer there. You either have to stop and restart FireFox, or alternately you can manually go to the above dialog on hit the Reload button to reload the script.

This process is not quite as smooth as it is with IE, but using the startup script is pretty easy as long as you remember to re-start FireFox to pick up the latest running status of Fiddler. I can live with that for debugging scenarios.

Getting Fiddler to work with .NET Client Requests

If you’ve run .NET client requests and tried to capture them with Fiddler you’ve probably noticed that .NET client requests don’t show up. This makes sense since proxy information must be configured somehow first in .NET as it doesn’t use the default settings that are provided through WinInet and the registry.

The scenario is as follows: You’re debugging a Web Service or a WebClient request in a client application or a test project and you would like to see what the HTTP response from the server looks like.

Most of the .NET HTTP clients – WebClient, HttpWebRequest, WCF or Web Service Clients – all have Proxy properties that you can set explicitly. In code this looks like this:

AppletService service = new AppletService();
service.Proxy = new WebProxy("127.0.0.1",8888);
WebServiceProxies.AppletService.Files[] files = service.RetrieveConfirmationList("0020575", "esa479");

Fortunately you don’t have to do this in code for every request. Instead you should preferrably use configuration file settings in the <system.net> section to configure the proxy:

<configuration>
  <system.net>
    <defaultProxy>
      <proxy  
              usesystemdefault="True"
              bypassonlocal="False" />
    </defaultProxy>
  </system.net>
</configuration>

The above reads the default system settings and uses them. So when Fiddler starts it sets the system settings and your application then can read and use these settings.

This should work fine for real client applications, however, if you’re running in a server application like ASP.NET you probably don’t want to do this because a registry lookup is required which can be really slow for a server application because it typically doesn’t have rights to access these settings.

If you need to look at HTTP client requests made by an ASP.NET application it’s best to configure the proxy explicitly by providing the proxy information in the web.config file and setting usesystemdefault=”False”

<configuration>
  <system.net>
    <defaultProxy>
      <proxy
              usesystemdefault="False"
              bypassonlocal="True" 
              proxyaddress="http://127.0.0.1:8888"              
              />
    </defaultProxy>
  </system.net>
</configuration>

This should let you see any Http requests fired from your server application i n Fiddler.

Make sure you enable All Processes in Fiddler

In Fiddler 2.x there was a change that allows filtering requests via a very unobvious status bar selection list. By default only browser requests can be tracked and all application requests are run through the proxy but don’t actually show up in Fiddler. To let your .NET applications (or any other application other than a browser including WinInet Clients) you need to make sure you set the filter to “All Applications”:

Clicking on the panel cycles through the options which is Web Browsers, Non-Browser, None and All Processes.

Thanks to Ryan Heaney who via Twitter pointed out this option which I completely overlooked and didn’t even find until he pointed it out. Talk about unobvious UI feature, although it’s actually nice ONCE you know where it is. It’s useful to keep other apps like feedreaders and twitter from posting to Fiddler when the setting is not set to All Processes.

The Underlying Connection was Closed

[section added 05/24/2009]

When running multiple authenticated requests with Fiddler in a .NET app you might see the following error:

The underlying connection was closed: A connection that was expected to be kept alive was closed by the server.

This is due to the fact that .NET by default closes connections after each requests, but Fiddler expects the connection to the proxy to be kept open (which is standard client proxy behavior). Typically what you’ll see is that the first request works, the next will fail with the error. If you retry the next will work, then fail again as the connection is closed on subsequent requests, re-opened when you retry.

There are two ways around this:

• Tell .NET to keep connections open
• Tweak the Fiddler Startup Script  to account for dropped connections

If you’re using  an HttpWebRequest object the process is fairly easy by adding adding headers to the request. You can set “Keep-Alive”,”On” or alternately explicitly force the connection to be closed with “Connection”,”close” which closes the connection completely.

WCF services can manipulate their underlying connection objects via the OperationContextScope() class, through which you can gain access to Http headers the client sends. Unfortunately, though the Connection and Keep-Alive headers cannot not be changed – WCF and only WCF controls those, so there’s no way to override the .NET behavior with these.

Luckily Fiddler is developed by a Microsoft developer who’s aware of this issue and so there’s actually a switch in the Fiddler configuration script that can be set to force Fiddler to automatically force connections closed. To do this open Fiddler’s CustomRules.js file (Rules | Customize Rules or Ctrl-R) and uncomment the following script block:

// Uncomment to reduce incidence of "unexpected socket closure" exceptions in .NET code.  
// Note that you really should also fix your .NET code to gracefully handle unexpected connection closure.
//
 if (!(((oSession.responseCode == 401) && oSession.oResponse["WWW-Authenticate"].Length > 9) || 
  ((oSession.responseCode == 407) && oSession.oResponse["Proxy-Authenticate"].Length > 9))) {
   oSession.oResponse["Connection"] = "close";
}

The above is commented in Fiddler's CustomRules.js  in the OnBeforeResponse method. Uncommenting the code above makes the successive Web Service calls work with Fiddler attached. As the comment mentions above this isn't ideal but it works - you may have to resort to a few additional conditions with NTLM Authentication as mentioned in this post:

http://blogs.newsgator.com/inbox/2006/09/fiddler_and_net.html

Monitoring HTTPS/SSL Connections

[added section 09/17/2009]

By default Fiddler doesn’t work with HTTPS/SSL requests which makes most people think that you can’t use it for SSL requests. It seems counterintuitive that an HTTP proxy should be able to decrypt SSL requests – after all it’s supposed to be secure and intractable, right? But this is possible because you’re explicitly installing a trusted certificate that can decrypt SSL traffic on your local machine.

It’s relatively straight forward to configure Fiddler to work with most SSL based sites (but not all of them if strict policy requirements are enabled on the server).

To use SSL you can enable HTTPS functionality in Tool | Options:

and follow the instructions in the Learn more… options.

You essentially have to export the Fiddler root and install it into the Root Certificate store. When you click the Decrypt Https checkbox a dialog will pop up asking whether you want to trust the Fiddler Root. If you do Fiddler will install the certificate into your root store for you and you’ll be able to decrypt HTTPS content that way.

Note that you probably should remove this certificate from the secure store once you’re done and add it only as needed. For more info see the Fiddler HTTPS page.

If Fiddler won’t work give Charles a Try

Fiddler is very nice when it works, but there are a few situations when I’ve had no luck with Fiddler. Specifically if you can’t explicitly set the Proxy in the client application such in a live server. When Fiddler doesn’t work or if it’s a pain to set up I fall back to Charles, which is Java based Http Proxy Monitor. It has many of the same features that Fiddler has, but it works much more transparently in that you don’t need to configure anything – Charles monitors any HTTP traffic on the wire and monitors it.

The first request above for example is a Cassini request and there’s no configuration of any sort to get Charles to monitor this request – it just works. You can view the request and response in a variety of ways. The view is not as nice as Fiddler’s split view but you can get the same information. One thing that Charles doesn’t do is auto-decode pages that are encoded with Gzip, which is kind of a bummer.

Charles is not free, but it’s worth the $50 registration to have a no-config Http Monitor that works in most environments. It’s saved my butt in a lot of situations…

Bottom line is that I still prefer Fiddler when it works because the UI is cleaner, easier to work with for many requests, easier to look at different kinds of contents and more configurable, but I keep around Charles if Fiddler doesn’t work or doesn’t work without complex configuration. In that respect Charles can be a life safer especially if you need to do some trouble shooting on a client or server machine in a hurry without having to fuck with configuration settings in the browser or configuration files.