Rick Strahl's Weblog  

Wind, waves, code and everything in between...
.NET • C# • Markdown • WPF • All Things Web
Contact   •   Articles   •   Products   •   Support   •   Advertise
Sponsored by:
West Wind WebSurge - Rest Client and Http Load Testing for Windows

ASP.NET Frameworks and Raw Throughput Performance


:P
On this page:

A few days ago I had a curious thought: With all these different technologies that the ASP.NET stack has to offer, what's the fastest technology overall to return raw data for a server request? When I started this it was mere curiosity rather than a real practical need or result. Different tools are used for different problems and so performance differences are to be expected. But still I was curious to see how the various technologies performed relative to each orher just for raw throughput of the request getting to the endpoint and back out to the client with as little processing in the actual endpoint logic as possible  (aka Hello World).

I want to clarify that this is merely an informal test for my own curiosity and I'm sharing the results and process here because I thought it was interesting. It's been a long while since I've done any sort of perf testing on ASP.NET, mainly because I've not had extremely heavy load requirements and because overall ASP.NET performs very well even for fairly high loads so that often it's not that critical to test for load performance. And that's a good thing!

This post is not meant to make a point  or even come to a conclusion which tech is better, but just to act as a reference to help understand some of the differences in perf and give a starting point to play around with yourself. I've included the code for this simple project, so you can play with it and maybe add a few additional tests for different things if you like:

I looked at these technologies:

  1. ASP.NET Web API
  2. ASP.NET MVC
  3. WebForms
  4. ASP.NET WebPages
  5. ASMX AJAX Services  (couldn't get AJAX/JSON to run on IIS8 )
  6. WCF Rest
  7. Raw ASP.NET HttpHandlers

It's quite a mixed bag, of course and the technologies target different types of development. What started out as mere curiosity turned into a bit of a head scratcher as the results were sometimes surprising.

First test: Raw Throughput

The first thing I tested was raw throughput for the various technologies. This is the least practical test of course since you're unlikely to ever create the equivalent of a 'Hello World' request in a real life application. The idea here is to measure how much time a 'NOP' request takes to return data to the client. So for this request I create the simplest Hello World request that I could come up for each tech.

Http Handler

The first is the lowest level approach which is an HTTP handler.

    public class Handler : IHttpHandler
    {
        public void ProcessRequest(HttpContext context)
        {
            context.Response.ContentType = "text/plain";
            context.Response.Write("Hello World. Time is: " + DateTime.Now.ToString());
        }

        public bool IsReusable
        {
            get
            {
                return true;
            }
        }
    }

WebForms

Next I added a couple of ASPX pages - one using CodeBehind and one using only a markup page.

The CodeBehind page simple does this in CodeBehind without any markup in the ASPX page:

    public partial class HelloWorld_CodeBehind : System.Web.UI.Page
    {
        protected void Page_Load(object sender, EventArgs e)
        {
            Response.Write("Hello World. Time is: " + DateTime.Now.ToString() );
            Response.End();
        }
    }

while the Markup page only contains some static output via an expression:

<%@ Page Language="C#" AutoEventWireup="false" 
         CodeBehind="HelloWorld_Markup.aspx.cs" 
        Inherits="AspNetFrameworksPerformance.HelloWorld_Markup" %>
Hello World. Time is <%= DateTime.Now %>

ASP.NET WebPages

WebPages is the freestanding Razor implementation of ASP.NET. Here's the simple HelloWorld.cshtml page:

Hello World @DateTime.Now

WCF REST

WCF REST was the token REST implementation for ASP.NET before WebAPI and the inbetween step from ASP.NET AJAX. I'd like to forget that this technology was ever considered for production use, but I'll include it here. Here's an OperationContract class:

    [ServiceContract(Namespace = "")]
    [AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
    public class WcfService
    {

        [OperationContract]
        [WebGet]
        public Stream HelloWorld()
        {
            var data = Encoding.Unicode.GetBytes("Hello World" + DateTime.Now.ToString());
            var ms = new MemoryStream(data);

            // Add your operation implementation here
            return ms;
        }
    }

WCF REST can return arbitrary results by returning a Stream object and a content type. The code above turns the string result into a stream and returns that back to the client.

ASP.NET AJAX (ASMX Services)

I also wanted to test ASP.NET AJAX services because prior to WebAPI this is probably still the most widely used AJAX technology for the ASP.NET stack today.

Unfortunately I was completely unable to get this running on my Windows 8 machine. Visual Studio 2012  removed adding of ASP.NET AJAX services, and when I tried to manually add the service and configure the script handler references it simply did not work - I always got a SOAP response for GET and POST operations. No matter what I tried I always ended up getting XML results even when explicitly adding the ScriptHandler. So, I didn't test this (but the code is there - you might be able to test this on a Windows 7 box).

ASP.NET MVC

Next up is probably the most popular ASP.NET technology at the moment: MVC. Here's the small controller:

    public class MvcPerformanceController : Controller
    {
        public ActionResult Index()
        {
            return View();
        }

        public ActionResult HelloWorldCode()
        {
            return new ContentResult() 
            { Content = "Hello World. Time is: " + DateTime.Now.ToString() };
        }
    }

ASP.NET WebAPI

Next up is WebAPI which looks kind of similar to MVC. Except here I have to use a StringContent result to return the response:

    public class WebApiPerformanceController : ApiController
    {
        [HttpGet]
        public HttpResponseMessage HelloWorldCode()
        {
            return new HttpResponseMessage() 
            { Content = new StringContent("Hello World. Time is: " + DateTime.Now.ToString(), Encoding.UTF8, "text/plain") };
        }
   }

Testing

Take a minute to think about each of the technologies… and take a guess which you think is most efficient in raw throughput. The fastest should be pretty obvious, but the others - maybe not so much.

The testing I did is pretty informal since it was mainly to satisfy my curiosity - here's how I did this: I used Apache Bench (ab.exe) from a full Apache HTTP installation to run and log the test results of hitting the server. ab.exe is a small executable that lets you hit a URL repeatedly and provides counter information about the number of requests, requests per second etc. ab.exe and the batch file are located in the \LoadTests folder of the project.

An ab.exe command line  looks like this:

ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorld

which hits the specified URL 100,000 times with a load factor of 20 concurrent requests. This results in output like this:

Console[4] 

It's a great way to get a quick and dirty performance summary. Run it a few times to make sure there's not a large amount of varience. You might also want to do an IISRESET to clear the Web Server. Just make sure you do a short test run to warm up the server first - otherwise your first run is likely to be skewed downwards. ab.exe also allows you to specify headers and provide POST data and many other things if you want to get a little more fancy. Here all tests are GET requests to keep it simple.

I ran each test:

  • 100,000 iterations
  • Load factor of 20 concurrent connections
  • IISReset before starting
  • A short warm up run for API and MVC to make sure startup cost is mitigated

Here is the batch file I used for the test:

IISRESET

REM make sure you add 
REM C:\Program Files (x86)\Apache Software Foundation\Apache2.2\bin
REM to your path so ab.exe can be found

REM Warm up
ab.exe -n100 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldJson
ab.exe -n100 -c20 http://localhost/aspnetperf/api/HelloWorldJson ab.exe -n100 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorld ab.exe -n100000 -c20 http://localhost/aspnetperf/handler.ashx > handler.txt ab.exe -n100000 -c20 http://localhost/aspnetperf/HelloWorld_CodeBehind.aspx > AspxCodeBehind.txt ab.exe -n100000 -c20 http://localhost/aspnetperf/HelloWorld_Markup.aspx > AspxMarkup.txt ab.exe -n100000 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorld > Wcf.txt ab.exe -n100000 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldCode > Mvc.txt ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorld > WebApi.txt

I ran each of these tests 3 times and took the average score for Requests/second, with the machine otherwise idle. I did see a bit of variance when running many tests but the values used here are the medians. Part of this has to do with the fact I ran the tests on my local machine - result would probably more consistent running the load test on a separate machine hitting across the network.

I ran these tests locally on my laptop which is a Dell XPS with quad core Sandibridge I7-2720QM @ 2.20ghz and a fast SSD drive on Windows 8. CPU load during tests ran to about 70% max across all 4 cores (IOW, it wasn't overloading the machine). Ideally you can try running these tests on a separate machine hitting the local machine. If I remember correctly IIS 7 and 8 on client OSs don't throttle so the performance here should be representative of what you can expect in a live scenario.

Results

Ok, let's cut straight to the chase. Below are the results from the tests…

Results[4]

The chart shows Requests per second for the different technologies. It's not surprising that the handler was fastest. But it was a bit surprising to me that the next fastest was WebForms and especially Web Forms with markup over a CodeBehind page. WebPages also fared fairly well. MVC and WebAPI are a little slower and the slowest by far is WCF REST (which again I find surprising).

As mentioned at the start the raw throughput tests are not overly practical as they don't test scripting performance for the HTML generation engines or serialization performances of the data engines. All it really does is give you an idea of the raw throughput for the technology from time of request to reaching the endpoint and returning minimal text data back to the client which indicates full round trip performance.

But it's still interesting to see that Web Forms performs better in throughput than either MVC, WebAPI or WebPages. It'd be interesting to try this with a few pages that actually have some parsing logic on it, but that's beyond the scope of this throughput test.

But what's also amazing about this test is the sheer amount of traffic that a laptop computer is handling. Even the slowest tech managed 5700 requests a second, which is one hell of a lot of requests if you extrapolate that out over a 24 hour period. Remember these are not static pages, but dynamic requests that are being served.

Another test - JSON Data Service Results

The second test I used a JSON result from several of the technologies. I didn't bother running WebForms and WebPages through this test since that doesn't make a ton of sense to return data from the them (OTOH, returning text from the APIs didn't make a ton of sense either :-)

In these tests I have a small Person class that gets serialized and then returned to the client. The Person class looks like this:

    public class Person
    {
        public Person()
        {
            Id = 10;
            Name = "Rick";
            Entered = DateTime.Now;
        }

        public int Id { get; set; }
        public string Name { get; set; }
        public DateTime Entered { get; set; }
    }

Here are the updated handler classes that use Person:

Handler

    public class Handler : IHttpHandler
    {
        public void ProcessRequest(HttpContext context)
        {
            var action = context.Request.QueryString["action"];
            if (action == "json")
                JsonRequest(context);
            else
                TextRequest(context);
        }

        public void TextRequest(HttpContext context)
        {
            context.Response.ContentType = "text/plain";
            context.Response.Write("Hello World. Time is: " + DateTime.Now.ToString());
        }

        public void JsonRequest(HttpContext context)
        {
            var json = JsonConvert.SerializeObject(new Person(), Formatting.None);
            context.Response.ContentType = "application/json";
            context.Response.Write(json);
        }

        public bool IsReusable
        {
            get
            {
                return true;
            }
        }
    }

This code adds a little logic to check for a action query string and route the request to an optional JSON result method. To generate JSON, I'm using the same JSON.NET serializer (JsonConvert.SerializeObject) used in Web API to create the JSON response.

WCF REST

 

    [ServiceContract(Namespace = "")]
    [AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
    public class WcfService
    {

        [OperationContract]
        [WebGet]
        public Stream HelloWorld()
        {
            var data = Encoding.Unicode.GetBytes("Hello World " + DateTime.Now.ToString());
            var ms = new MemoryStream(data);

            // Add your operation implementation here
            return ms;
        }
        
        [OperationContract]
        [WebGet(ResponseFormat=WebMessageFormat.Json,BodyStyle=WebMessageBodyStyle.WrappedRequest)]
        public Person HelloWorldJson()
        {
            // Add your operation implementation here
            return new Person();
        }

    }

For WCF REST all I have to do is add a method with the Person result type.

 

ASP.NET MVC

    public class MvcPerformanceController : Controller
    {
        //
        // GET: /MvcPerformance/

        public ActionResult Index()
        {
            return View();
        }

        public ActionResult HelloWorldCode()
        {
            return new ContentResult() 
            { Content = "Hello World. Time is: " + DateTime.Now.ToString() };
        }

        public JsonResult HelloWorldJson()
        {
            return Json(new Person(), JsonRequestBehavior.AllowGet);
        }

    }

For MVC all I have to do for a JSON response is return a JSON result. ASP.NET internally uses JavaScriptSerializer.

ASP.NET WebAPI

    public class WebApiPerformanceController : ApiController
    {
        [HttpGet]
        public HttpResponseMessage HelloWorldCode()
        {
            return new HttpResponseMessage() 
            { Content = new StringContent("Hello World. Time is: " + DateTime.Now.ToString(), Encoding.UTF8, "text/plain") };
        }

        [HttpGet]
        public Person HelloWorldJson()
        {
            return new Person();
        }

        [HttpGet]
        public HttpResponseMessage HelloWorldJson2()
        {
            var response = new HttpResponseMessage(HttpStatusCode.OK);
            response.Content = new ObjectContent<Person>(new Person(),
                            GlobalConfiguration.Configuration.Formatters.JsonFormatter);
            return response;
        }

   }

Testing and Results

To run these data requests I used the following ab.exe commands:

REM JSON RESPONSES
ab.exe -n100000 -c20 http://localhost/aspnetperf/Handler.ashx?action=json > HandlerJson.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/MvcPerformance/HelloWorldJson > MvcJson.txt
ab.exe -n100000 -c20 http://localhost/aspnetperf/api/HelloWorldJson > WebApiJson.txt 
ab.exe -n100000 -c20 http://localhost/AspNetPerf/WcfService.svc/HelloWorldJson > WcfJson.txt

The results from this test run are a bit interesting in that the WebAPI test improved performance significantly over returning plain string content.

Here are the results:

Results2[4] 

The performance for each technology drops a little bit except for WebAPI which is up quite a bit! From this test it appears that WebAPI is actually significantly better performing returning a JSON response, rather than a plain string response.

Snag with Apache Benchmark and 'Length Failures'

I ran into a little snag with Apache Benchmark, which was reporting failures for my Web API requests when serializing. As the graph shows performance improved significantly from with JSON results from 5580 to 6530 or so which is a 15% improvement (while all others slowed down by 3-8%).

However, I was skeptical at first because the WebAPI test reports showed a bunch of errors on about 10% of the requests. Check out this report:

ResultsCommand2

Notice the Failed Request count. What the hey? Is WebAPI failing on roughly 10% of requests when sending JSON?

Turns out: No it's not! But it took some sleuthing to figure out why it reports these failures. At first I thought that Web API was failing, and so to make sure I re-ran the test with Fiddler attached and runiisning the ab.exe test by using the -X switch:

ab.exe -n100 -c10 -X localhost:8888 http://localhost/aspnetperf/api/HelloWorldJson

which showed that indeed all requests where returning proper HTTP 200 results with full content.

However ab.exe was reporting the errors. After some closer inspection it turned out that the dates varying in size altered the response length in dynamic output.

For example: these two results:

{"Id":10,"Name":"Rick","Entered":"2012-09-04T10:57:24.841926-10:00"}

{"Id":10,"Name":"Rick","Entered":"2012-09-04T10:57:24.8519262-10:00"}

are different in length for the number which results in 68 and 69 bytes respectively. The same URL produces different result lengths which is what ab.exe reports. I didn't notice at first bit the same is happening when running the ASHX handler with JSON.NET result since it uses the same serializer that varies the milliseconds.

Moral: You can typically ignore Length failures in Apache Benchmark and when in doubt check the actual output with Fiddler. Note that the other failure values are accurate though.

Another interesting Side Note: Perf drops over Time

As I was running these tests repeatedly I was finding that performance steadily dropped from a startup peak to a 10-15% lower stable level. IOW, with Web API I'd start out with around 6500 req/sec and in subsequent runs it keeps dropping until it would stabalize somewhere around 5900 req/sec occasionally jumping lower. For these tests this is why I did the IIS RESET and warm up for individual tests. This is a little puzzling. Looking at Process Monitor while the test are running memory very quickly levels out as do handles and threads, on the first test run. Subsequent runs everything stays stable, but the performance starts going downwards. This applies to all the technologies - Handlers, Web Forms, MVC, Web API - curious to see if others test this and see similar results.

Doing an IISRESET then resets everything and performance starts off at peak again…

Summary

As I stated at the outset, these were informal to satiate my curiosity not to prove that any technology is better or even faster than another. While there clearly are differences in performance the differences (other than WCF REST which was by far the slowest and the raw handler which was by far the highest) are relatively minor, so there is no need to feel that any one technology is a runaway standout in raw performance. Choosing a technology is about more than pure performance but also about the adequateness for the job and the easy of implementation. The strengths of each technology will make for any minor performance difference we see in these tests.

However, to me it's important to get an occasional reality check and compare where new technologies are heading. Often times old stuff that's been optimized and designed for a time of less horse power can utterly blow the doors off newer tech and simple checks like this let you compare. Luckily we're seeing that much of the new stuff performs well even in V1.0 which is great.

To me it was very interesting to see Web API perform relatively badly with plain string content, which originally led me to think that Web API might not be properly optimized just yet. For those that caught my Tweets late last week regarding WebAPI's slow responses was with String content which is in fact considerably slower. Luckily where it counts with serialized JSON and XML WebAPI actually performs better. But I do wonder what would make generic string content slower than serialized code?

This stresses another point: Don't take a single test as the final gospel and don't extrapolate out from a single set of tests. Certainly Twitter can make you feel like a fool when you post something immediate that hasn't been fleshed out a little more <blush>. Egg on my face. As a result I ended up screwing around with this for a few hours today to compare different scenarios. Well worth the time…

I hope you found this useful, if not for the results, maybe for the process of quickly testing a few requests for performance and charting out a comparison. Now onwards with more serious stuff…

Resources

Posted in ASP.NET  Web Api  

The Voices of Reason


 

Bertrand Le Roy
September 04, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

When I saw your test, I knew that ASP.NET WebForms would perform very well. Having done some work back in the days on the ProcessContentMain method that is doing most of the work, I know that the hello world scenario is optimized to death. What you are doing here is very close to the internal test case that is used to prevent perf regressions on the core pipeline. The problem is that it's not a realistic scenario, you just hit a super-optimized part of the framework. Just do something as trivial as reading a query string parameter and weep for your perf. I predict a 10-20% drop just for that, if the code in there hasn't changed too much.
I suspect all the other tech that seems not to perform as well has been optimized for realistic scenarios rather than for this silly case. This proves one thing: this team learns from its mistakes ;)

Rick Strahl
September 04, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

@Bertrand - you mean inside of the actual markup page? But doesn't the page compile down to code anyway - it's not completely static in that the date is an expression (I knew there's major optimization to fully static ASPX pages). I just tried adding a code block to it with a Request.QueryString read in there and the performance didn't drop much, so I think the special case might just be purely static pages without any expression - any expression requires compiling down the page into a class I would think.

But your point is well taken anyway - it's not super realistic since the processing overhead is probably the smallest part of most requests - database access and other computations eat up way more cycles than the request routing most likely in any app that does real work. Still it's useful to compare sometimes :-)

Bertrand Le Roy
September 04, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

No, it's not compilation but rather that particular code path that has been extremely optimized. Then again, I'm saying this based on the state of that code as I knew it back when I was a dev on the team, and enough years have passed that further optimizations, for example in the parsing of the query string, may have been applied as well since then. But yeah, that's the idea, benchmark real use cases rather than hello worlds... ;)

Daniel
September 05, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

Would be interesting to see how other frameworks compare to this? E.g NancyFx and ServiceStack.

//Daniel

Rick Strahl
September 05, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

@Daniel - Code's available online. Fork the repo and add it... Be cool to get a wider range of frameworks in there.

Fujiy
September 05, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

The Perf drops over Time could be in reason of Garbage Collector

Mitch Labrador
September 05, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

Very interesting to see. First comparison of this nature I've seen so far. Lesson, steer away from WCF :P

Joe Brinkman
September 05, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

Like others, I don't place much stock in such simplistic tests as the results will likely be drastically different in real world scenarios. What I do find interesting is seeing how much ceremony or effort is required for simple tasks.

Darren Kopp
September 05, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

How was your system set up? I assume you compiled release build, but did you change machine.config to be <deployment retail="true" /> ?

Scott Holodak
September 05, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

Now that most of the newer technologies are moving towards open source/out-of-band with VS and .NET Framework, at least we won't have to wait 2 years for the performance to improve and the kinks to get worked out.

David Robbins
September 05, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

Rick, regarding your experience with Asp.net AJAX and VS 2012 are you of the opinion that asmx services were removed so that it encourage more adoption of WCF? I use ASMX services for a lot of my development, having found WCF too complex and would hate to think that in VS 2012 it's too difficult to be practical.

Chris Marisic
September 05, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

I'd love to see the results of these test running against Server 2012 and the parallelization increased from 20 concurrent requests to 500 or 1000. I have a feeling the relative performance results will be substantially different.

FickePanther
September 05, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

Interesting results! Personally I agree with what you said at the end. It's not always about the performance as what is right for the job. Personally I got sick and tired with developers fighting with web forms trying to get something that should be simple to work. I would take lower performance but save developers pain and trouble any day. Then again performance where I work is not as important as "does it work" right now.

Thanks for the metrics though! Pretty cool idea to try out.

moody1987
September 06, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

I had been doing similar performance tests on my own but the results were clearly different. The throughput of 6-7k req/s for asp.net MVC is quite convincing. You can find on the web many blog entries from people that had similar results.

But I am surprised about webforms performance. My tests results showed that you can easily achieve about 12-13k req/s for simple HelloWorld webforms application.

Did you check performance counters on the tested machine? Maybe your machine performance was a bottleneck and 8k req/s is the upper limit for this server.

Rick Strahl
September 06, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

@moody - I don't think so, because Handlers top out at 7.5k or so and everything else falls roughly where I would expect it to (ie. handler clearly highest everything else below). When the machine gets loaded up a bit more (starts slowing down) the divergence percentages stay the same. If there was a bottleneck with connections we'd top out at a top connection value I would think.

I think it depends on the hardware obviously, although this is a very fast machine for a laptop. I'm curious to see what experiences other people have with results for given hardware.

Samuel
September 09, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

Hi Rick,

interesting tests. For ASMX, have you thought about testing PageMethods instead (static methods inside codebehind files)? I don't have IIS8 to test.

They are quite similar and I use PageMethods almost exclusively in place of ASMX to return json to jQuery -> I don't have to compile it which is a big time saver.

Rick Strahl
September 09, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

@Samuel - I didn't test any of the Script stuff in ASMX because I couldn't get it to work on Windows 8. It always returned XML regardless of whether methods where marked up. PageMethods use the same mechanism as ASMX AJAX requests and roughly get the same performance since they're not actually running through the page pipeline since all the methods are static.

Jomit
September 10, 2012

# re: ASP.NET Frameworks and Raw Throughput Performance

I agree with Joe. I think that once we add the data access and other layers these results may change drastically. Also I think that if these tests were done on a Server OS with throttling than WCF would have better results.

Nice article..

Eamon Nerbonne
February 04, 2013

# re: ASP.NET Frameworks and Raw Throughput Performance

Some context would be nice here; e.g. the number of static pages IIS can serve.

Looks to me like all these measurements are so close as to be equivalent. If even such a pure-overhead scenario shows just a 40% difference, as soon are you do any actual work you'll likely swamp this anyhow.

Markus
December 05, 2013

# re: ASP.NET Frameworks and Raw Throughput Performance

WCF REST part should be tested with following code, if you are going to test only performance:
[OperationContract]
[WebGet]
public string HelloWorld()
{
return "Hello World " + DateTime.Now.ToString();
}

[OperationContract]
[WebGet(ResponseFormat=WebMessageFormat.Json,BodyStyle=WebMessageBodyStyle.Bare)]
public Person HelloWorldJson()
{
// Add your operation implementation here
return new Person();
}

Rick Strahl
December 06, 2013

# re: ASP.NET Frameworks and Raw Throughput Performance

@Markus - you can't return RAW data with WCF rest unless you return a stream. That's the point. If you look further down you see an example of returning JSON, but if you want to return a non-JSON value or string then a Stream result is the only way.

Dmitriy@itadapter
April 06, 2014

# re: ASP.NET Frameworks and Raw Throughput Performance

Rick, you forgot the "-k" keep-alive switch.
Interestingly, I get 9,800 req/sec between my laptop and server under the desk, but as soon as I stick "-k" I get 85,000 req/sec.

ab -n100000 -c36 -k "http://127.0.0.1:8080/embed/count?from=1&to=1"

I run HttpListener-based server (no IIS) and completely bypass MS web stack, but that is besides the point

Nikolai
July 24, 2014

# re: ASP.NET Frameworks and Raw Throughput Performance

Thanks for doing this bechmark and amazing results and summary. It was really good to see that ASPX doesn't perform much worse than ASHX.

There's a chance that performance drop over time may be attributed to turbo-boost which is on your 2720QM CPU unless you disable it in BIOS.

The way it works is, once CPU is loaded, it will speed up to 3.3GHz instead of 2.2GHz. However as the CPU potentially gets hotter during the test, there's a chance that it starts to automatically clock down to produce less heat.

Quad-core mobile CPUs are the worst in that aspect as they are in a bucket of design power 45W as opposed to 35W for dual-cores.

What you can do is install a free utility CPUz that will show you CPU speed while you run bechmarks. Or, considering it is Dell, you can actually disable turbo-boost in BIOS just as it seems you already disable Hyper-Threading (which is good to do for consistency of your test). There's also another free tool SpeedFan that can show you core temperatures while you run the test. What you may notice is that just few seconds after you stop the test to IISRESET, core temperatures may drop from ~ 70-80 degrees to ~ 50-60 degrees, thus bumping turbo boost back to max.

Andy
February 27, 2015

# re: ASP.NET Frameworks and Raw Throughput Performance

I've always referred to this article over the years in my various discussions of performance :) Any chance you might be able to revisit it for the new 2015 technologies (MS has put a lot of effort into performance since 2012)

Yassine
March 05, 2015

# re: ASP.NET Frameworks and Raw Throughput Performance

Thank you for this great article.

I agree with Andy, would vNext be faster ? ( because they have merged Web api and Mvc, and is more lighter because it doesn't reference System.Web and the heavy 200mb framework ? ( lighter means small memory print to handle each request... ) )

These are good questions to answer :) .

Thanks again

Rick Strahl
March 05, 2015

# re: ASP.NET Frameworks and Raw Throughput Performance

Currently the answer is no. vNext is very slow compared to say Web API or current MVC, but it's also not optimized yet. In my tests I ran in with beta 2 perf was somewhere in the 75% range from what MVC/Web API produced.

I haven't added vNext to the tests because of the unstable and changing environment and the really crappy perf at the moment. This will get better once they get closer to a fixed set of features and release candidates I think.

To be honest I don't expect performance to be greatly improved over current tech, but scalability - higher overall request load might be the case if code is properly async optimized.

We'll just have to wait.

Spencer
April 12, 2015

# re: ASP.NET Frameworks and Raw Throughput Performance

Great article. i pulled down this code to do some comparisons on web api performance. I was seeing insanely slow RPS of around 170 on my work laptop using my code with the service hosted locally. So i downloaded this and didn't see much of an improvement. I then put this same code on my desktop at home which is running a i7-2600K 3.4Ghz quad core with 24 gigs of ram. the cpu never gets up over 60% but the RPS only hits around 600 RPS using anywhere from 20 to 100 concurrent users and the hosting type never seem to make much difference. I know i've seen multi thousand RPS numbers when building old asmx services but i'm scratching my head on whats causing the issue now. It seems like there is some unusual thread contention going on but i haven't dug too deep yet. Any ideas?

Rick Strahl
April 12, 2015

# re: ASP.NET Frameworks and Raw Throughput Performance

@Spencer, it sounds like you're hitting IO limitations (CPU not maxed). What are you doing in your own tests? If you hit a database as part of testing your request/sec is going to drop drastically.

So I would first set up basic helloworld requests and test those just to see the max throughput you could possibly expect.

Also make sure to turn of Anti-Virus/Firewall software while running these tests. They can drastically slow down and throttle requests especially if you run to custom non-80 ports.

Spencer
April 16, 2015

# re: ASP.NET Frameworks and Raw Throughput Performance

The services we are building are async and have db calls within them, but the dbs run on dedicated servers separately, so there is always a bit of waiting in them. I was basically just doing some simple test runs to get a basic idea of how many servers we were going to need to support N users simultaneously. When i hit the snag with the real services i setup some cookie cutter ones including using your test solution to see what the raw performance would be with minimal dependencies.

I thought i turned off the anti-virus, but i guess it had turned itself back on my personal box. :-) The numbers are now hitting around 6k for the raw service calls on my personal box. The interesting thing is i don't remember seeing anti virus software running on our work server but there must be something running that is not obvious and was not showing itself in the task manager. Definitely the firewall was off but i'll have to do a bit more digging on that. Now at least i know i'm not crazy since i knew i'd seen higher numbers in the past. Thanks again for your input.

Dmitriy
May 05, 2015

# re: ASP.NET Frameworks and Raw Throughput Performance

Please add "-k" to your tests. You are showing results which are 5-6 times lower than they atually are

Matt Watson
November 27, 2015

# re: ASP.NET Frameworks and Raw Throughput Performance

When testing performance, be sure to use an application performance monitoring (APM) tool so you can track page load times, slow database queries, etc for every request in your app. You can also use them to see interactions with web services, caching, queues, etc. They are really invaluable tools for performance testing and tuning!

Thanks,
Matt Watson with Stackify APM (http://www.stackify.com)

West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2024