Rick Strahl's Web Log

Wind, waves, code and everything in between...
ASP.NET • C# • HTML5 • JavaScript • AngularJs
Contact   •   Articles   •   Products   •   Support   •   Search
Ad-free experience sponsored by:
ASPOSE - the market leader of .NET and Java APIs for file formats – natively work with DOCX, XLSX, PPT, PDF, images and more

Using FiddlerCore to capture HTTP Requests with .NET


Over the last few weeks I’ve been working on my Web load testing utility West Wind WebSurge. One of the key components of a load testing tool is the ability to capture URLs effectively so that you can play them back later under load. One of the options in WebSurge for capturing URLs is to use its built-in capture tool which acts as an HTTP proxy to capture any HTTP and HTTPS traffic from most Windows HTTP clients, including Web Browsers as well as standalone Windows applications and services.

To make this happen, I used Eric Lawrence’s awesome FiddlerCore library, which provides most of the functionality of his desktop Fiddler application, all rolled into an easy to use library that you can plug into your own applications. FiddlerCore makes it almost too easy to capture HTTP content!

For WebSurge I needed to capture all HTTP traffic in order to capture the full HTTP request – URL, headers and any content posted by the client. The result of what I ended up creating is this semi-generic capture form:

CaptureForm

In this post I’m going to demonstrate how easy it is to use FiddlerCore to build this HTTP Capture Form. 

If you want to jump right in here are the links to get Telerik’s Fiddler Core and the code for the demo provided here.

Note that FiddlerCore is bound by a license for commercial usage – see license.txt in the FiddlerCore distribution for details.

Integrating FiddlerCore

FiddlerCore is a library that simply plugs into your application. You can download it from the Telerik site and manually add the assemblies to your project, or you can simply install the NuGet package via:

      PM> Install-Package FiddlerCore

The library consists of the FiddlerCore.dll as well as a couple of support libraries (CertMaker.dll and BCMakeCert.dll) that are used for installing SSL certificates. I’ll have more on SSL captures and certificate installation later in this post.

But first let’s see how easy it is to use FiddlerCore to capture HTTP content by looking at how to build the above capture form.

Capturing HTTP Content

Once the library is installed it’s super easy to hook up Fiddler functionality. Fiddler includes a number of static class methods on the FiddlerApplication object that can be called to hook up callback events as well as actual start monitoring HTTP URLs.

In the following code directly lifted from WebSurge, I configure a few filter options on Form level object, from the user inputs shown on the form by assigning it to a capture options object. In the live application these settings are persisted configuration values, but in the demo they are one time values initialized and set on the form. Once these options are set, I hook up the AfterSessionComplete event to capture every URL that passes through the proxy after the request is completed and start up the Proxy service:

void Start()
{
    if (tbIgnoreResources.Checked)
        CaptureConfiguration.IgnoreResources = true;
    else
        CaptureConfiguration.IgnoreResources = false;

    string strProcId = txtProcessId.Text;
    if (strProcId.Contains('-'))
        strProcId = strProcId.Substring(strProcId.IndexOf('-') + 1).Trim();

    strProcId = strProcId.Trim();

    int procId = 0;
    if (!string.IsNullOrEmpty(strProcId))
    {
        if (!int.TryParse(strProcId, out procId))
            procId = 0;
    }
    CaptureConfiguration.ProcessId = procId;
    CaptureConfiguration.CaptureDomain = txtCaptureDomain.Text;

    FiddlerApplication.AfterSessionComplete += FiddlerApplication_AfterSessionComplete;
    FiddlerApplication.Startup(8888, true, true, true);
}

The key lines for FiddlerCore are just the last two lines of code that include the event hookup code as well as the Startup() method call. Here I only hook up to the AfterSessionComplete event but there are a number of other events that hook various stages of the HTTP request cycle you can also hook into. Other events include BeforeRequest, BeforeResponse, RequestHeadersAvailable, ResponseHeadersAvailable and so on.

In my case I want to capture the request data and I actually have several options to capture this data. AfterSessionComplete is the last event that fires in the request sequence and it’s the most common choice to capture all request and response data. I could have used several other events, but AfterSessionComplete is one place where you can look both at the request and response data, so this will be the most common place to hook into if you’re capturing content.

The implementation of AfterSessionComplete is responsible for capturing all HTTP request headers and it looks something like this:

private void FiddlerApplication_AfterSessionComplete(Session sess)
{
    // Ignore HTTPS connect requests
    if (sess.RequestMethod == "CONNECT")
        return;

    if (CaptureConfiguration.ProcessId > 0)
    {
        if (sess.LocalProcessID != 0 && sess.LocalProcessID != CaptureConfiguration.ProcessId)
            return;
    }

    if (!string.IsNullOrEmpty(CaptureConfiguration.CaptureDomain))
    {
        if (sess.hostname.ToLower() != CaptureConfiguration.CaptureDomain.Trim().ToLower())
            return;
    }

    if (CaptureConfiguration.IgnoreResources)
    {
        string url = sess.fullUrl.ToLower();

        var extensions = CaptureConfiguration.ExtensionFilterExclusions;
        foreach (var ext in extensions)
        {
            if (url.Contains(ext))
                return;
        }

        var filters = CaptureConfiguration.UrlFilterExclusions;
        foreach (var urlFilter in filters)
        {
            if (url.Contains(urlFilter))
                return;
        }
    }

    if (sess == null || sess.oRequest == null || sess.oRequest.headers == null)
        return;

    string headers = sess.oRequest.headers.ToString();
    var reqBody = sess.GetRequestBodyAsString();

    // if you wanted to capture the response
    //string respHeaders = session.oResponse.headers.ToString();
    //var respBody = session.GetResponseBodyAsString();

    // replace the HTTP line to inject full URL
    string firstLine = sess.RequestMethod + " " + sess.fullUrl + " " + sess.oRequest.headers.HTTPVersion;
    int at = headers.IndexOf("\r\n");
    if (at < 0)
        return;
    headers = firstLine + "\r\n" + headers.Substring(at + 1);

    string output = headers + "\r\n" +
                    (!string.IsNullOrEmpty(reqBody) ? reqBody + "\r\n" : string.Empty) +
                    Separator + "\r\n\r\n";

    BeginInvoke(new Action<string>((text) =>
    {
        txtCapture.AppendText(text);
        UpdateButtonStatus();
    }), output);

}

The code starts by filtering out some requests based on the CaptureOptions I set before the capture is started. These options/filters are applied when requests actually come in. This is very useful to help narrow down the requests that are captured for playback based on options the user picked. I find it useful to limit requests to a certain domain for captures, as well as filtering out some request types like static resources – images, css, scripts etc. This is of course optional, but I think it’s a common scenario and WebSurge makes good use of this feature.

AfterSessionComplete like other FiddlerCore events, provides a Session object parameter which contains all the request and response details. There are oRequest and oResponse objects to hold their respective data. In my case I’m interested in the raw request headers and body only, as you can see in the commented code you can also retrieve the response headers and body. Here the code captures the request headers and body and simply appends the output to the textbox on the screen. Note that the Fiddler events are asynchronous, so in order to display the content in the UI they have to be marshaled back the UI thread with BeginInvoke, which here simply takes the generated headers and appends it to the existing textbox test on the form. As each request is processed, the headers are captured and appended to the bottom of the textbox resulting in a Session HTTP capture in the format that Web Surge internally supports, which is basically raw request headers with a customized 1st HTTP Header line that includes the full URL rather than a server relative URL.

When the capture is done the user can either copy the raw HTTP session to the clipboard, or directly save it to file. This raw capture format is the same format WebSurge and also Fiddler use to import/export request data.

While this code is application specific, it demonstrates the kind of logic that you can easily apply to the request capture process, which is one of the reasonsof why FiddlerCore is so powerful. You get to choose what content you want to look up as part of your own application logic and you can then decide how to capture or use that data as part of your application.

The actual captured data in this case is only a string. The user can edit the data by hand or in the the case of WebSurge, save it to disk and automatically open the captured session as a new load test.

Posted in .NET  HTTP  

The Voices of Reason


 

Eric Lawrence
July 29, 2014

# re: Using FiddlerCore to capture HTTP Requests with .NET

Hey, Rick-- Thanks for an awesome post!

If you're trying to get the request or response body as a string, you should use the oSession.GetRequestBodyAsString() and oSession.GetResponseBodyAsString() methods respectively. It's not safe to assume that all textual bodies are UTF-8 encoded; these methods will examine the charset declaration in the headers and/or the body when decoding the bytes to a string.

You can get the Fiddler book as a PDF; see http://fiddlerbook.com

One final caveat: While Fiddler is free for all and Telerik has committed to it remaining so indefinitely, FiddlerCore is provided under a different license with open source and commercial terms. FiddlerCore's license can be found in the installation package.

Thanks again!

Rick Strahl
July 29, 2014

# re: Using FiddlerCore to capture HTTP Requests with .NET

@Eric - Thanks. I've updated the post with GetRequestBodyAsString() - also my code :-). I also added a blurp about licensing.

Jamy Ryals
August 01, 2014

# re: Using FiddlerCore to capture HTTP Requests with .NET

Beware, you need a lawyer to understand the license for FiddlerCore. I have no idea if I'm able to use it or not.

Solomon
October 20, 2014

# re: Using FiddlerCore to capture HTTP Requests with .NET

Great work -request to server normally come from client anywhere not localhost. Hostname is confiured to this regards.Am I missing anything here please - do this program decrypt SSL from incoming request from another machine even though certificates was created in the code?. I can see this in code // Ignore HTTPS connect requests
if (sess.RequestMethod == "CONNECT")
return;

Rick Strahl
October 20, 2014

# re: Using FiddlerCore to capture HTTP Requests with .NET

@Solomon - did you actually read the post??? You can obviously capture output from other domains but you can capture localhost traffic. You can also capture SSL traffic but you have to enable it - see the bottom section of the post.

Ted Yang
November 27, 2014

# re: Using FiddlerCore to capture HTTP Requests with .NET

Hi Rick,

really great topic about using fiddlerCore to capture traffic, I have learned a lot. I'm working on a testing project which use selenium to automate our web test. I also want to use fiddlerCore to capture the traffic and make everything automated. Everything goes well except the "InstallCertificate()", when CertMaker.trustRootCert(), it will popup a window and ask if you want to trust this cert. Do you know there is a way to make this action silent?

Best wishes,
Ted

Rick Strahl
November 27, 2014

# re: Using FiddlerCore to capture HTTP Requests with .NET

@Ted - One time when the cert is created you get the dialog but after this not anymore. So I suggest you add the certificate to your machine, then your test should just work with SSL.

Code in the article shows you how you can check whether the cert is already installed and how to install it.

Ted Yang
November 30, 2014

# re: Using FiddlerCore to capture HTTP Requests with .NET

Hi Rick,

Thank you for your reply. Yes, as you said it's one time, after I install the cert with makecert.exe, it works. My problem is that our team make the test run on multiple robots, and the robots could auto scale out, then the new created VMs may not have the cert. That's why I want to make everything automated.

Rick Strahl
November 30, 2014

# re: Using FiddlerCore to capture HTTP Requests with .NET

@Ted - you can't auto-install certificates. This is a Windows feature that requires user authorization to prevent malicious installation of man in the middle certificates. So you just have to pre-install the certificates on your test servers.

Chris
December 03, 2014

# re: Using FiddlerCore to capture HTTP Requests with .NET

Rick, thanks for the great post, it really helped me. Eric, thanks for the awesome tool. I'm wondering if there is a place I could learn more about the timer? I'm specifically wanting to know how long it takes to 'get a connected to the server' and how long it takes to 'download the file after the connection is established'. In quotes because I'm sure my terminology is incorrect. In other words, which timer objects I would use.

Ira
March 12, 2015

# re: Using FiddlerCore to capture HTTP Requests with .NET

Greate job! thank you!
but I have 1 trouble.
in my local machine Fiddler creates certificate and FiddlerCore Api works perfectly.

but I need to build my project also on CI TeamCity and there I have some problem.
FiddlerCore could not create certificate.
my code fails on this line:
"CertMaker.createRootCert()"
with next error message:
"System.IO.FileNotFoundException : Cannot locate: MakeCert.exe. Please move makecert.exe to the Fiddler installation directory.
at Fiddler.DefaultCertificateProvider.CreateCert(String sHostname, Boolean isRoot)
at Fiddler.DefaultCertificateProvider.CreateRootCertificate()".

In my project I have 2 dlls: BCMakeCert.dll and CertMaker.dll;

and here is my method:
"void InstallCertificate()
{
if (!CertMaker.rootCertExists())
{
if (!CertMaker.createRootCert())
{
throw new Exception("Unable to create certificate for FiddlerCore.");
}
if (!CertMaker.trustRootCert())
{
throw new Exception("Unable to trust certificate for FiddlerCore.");
}

X509Store certStore = new X509Store(StoreName.Root, StoreLocation.LocalMachine);
certStore.Open(OpenFlags.ReadWrite);
try
{
certStore.Add(CertMaker.GetRootCertificate());
}
finally
{
certStore.Close();
}
}
}"

where I miss something??

thank you a lot!

Ravikanth
October 12, 2015

# re: Using FiddlerCore to capture HTTP Requests with .NET

hello every,
acutally i wanted to get the the body iam using var reqBodystring = sess.GetRequestBodyAsString(); iam not getting the body content correct it is encrypted format how can i decrypt it and get the body content,can some body help.

Amit
February 05, 2017

# re: Using FiddlerCore to capture HTTP Requests with .NET

In fiddler core's new versions (4.6.3.50306) i can't able to get all the preferences (especially "fiddler.certmaker.bc.cert" & "fiddler.certmaker.bc.key") but in the old version (i.e. 4.5.1) its working fine.

Code: FiddlerApplication.Prefs.GetStringPref("fiddler.certmaker.bc.cert", null));

FiddlerApplication.Prefs.GetStringPref("fiddler.certmaker.bc.key", null));

What might be the problem?


Gopi V
December 06, 2017

# re: Using FiddlerCore to capture HTTP Requests with .NET

Hi Rick, Is it possible to use FC (fiddler core) to pause incoming IIS requests? when PIC goes very high with cpu, I'd like to pause the incoming requests, monitor PIC to come down and let the requests get in or timeout the requets... (like IIS 6 Pause function, seems IIS 7+ removed it) If you know any 3rd party tool provides such functionality would also be useful.

Regards

Gopi

 

West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2017