Rick Strahl's Weblog  

Wind, waves, code and everything in between...
.NET • C# • Markdown • WPF • All Things Web
Contact   •   Articles   •   Products   •   Support   •   Advertise
Sponsored by:
West Wind WebSurge - Rest Client and Http Load Testing for Windows

Re-examining ASP.NET and Helios Performance Tests


:P
On this page:

Since I got a new machine (Dell XPS 15) last week  I decided to re-run my Helios load tests along with a few other usage scenarios and was pleasantly surprised by the improved performance which is boosted by over 30%, but also found some odd behavior in some of the Web API related load tests. This post is kind of a rambling review of some of the stuff I found – rambling because I’m not quite sure how to account for some of the results.

So, last year I posted a blog entry related to ASP.NET raw throughput performance which aimed to measure basic request throughput for the various ASP.NET frameworks available today. A couple of weeks ago I added Helios to this and found that Helios improved performance quite a bit. Last week I also got a new machine and as I set up the machine I re-ran some of the tests I had previously run and found that performance with this new Dell XPS 15 laptop is near over 30% faster than my previous XPS 15 (both of which have quad-core cpu’s rated at the same speed and the same SSD as before).

Here are the numbers I ended up with:

ApacheBench[1]

The numbers are not all that different than my original Helios tests – except faster. The spread between technologies is staying roughly in the same range. But for a laptop these numbers are pretty amazing. I ran these tests about 20 different times and while the overall top values varied a bit the relationship between the different technologies staid always just about constant.

I also updated the parser to show errors – note that there are a bunch of errors for the JSON results, which is actually OK. AB.exe which I use for the load tests detects that the response is not the identical length and flags the request. This is normal as the result contains a JSON date which varies in length depending on the millisecond count.

Anti-Virus Seriously Affects Performance

Even before I upgraded to the new machine, by chance turned off Windows Defender and retested performance. I was having some odd issues in another Web application where files were mysteriously disappearing so I turned off Defender. While I had it off I re-ran the ASP.NET Perf tests – performance immediately jumped by 30%! The numbers above reflect the 30% Defender penalty removal, plus another 30% on top of that from the numbers I published in the original performance article. This is a significant performance hit to take and AFAIK Windows Defender is one of the less intrusive/hogging AV applications.

I suppose on a desktop machine this hit is not a serious problem but I on a server it might be worthwhile to see if server AV has a similar effect and if it does tweak the AV to not affect network traffic so heavily.

Web API

For those of you on Twitter you might have seen some Web API related posts from me over the weekend. While testing on the new machine, I mentioned some numbers and a few people chimed in with Web API questions, like how are you returning results etc. So I added a few more tests to check for returning results in a few different ways.

  • Returning a Typed Result
  • Return HttpResponseMessage created with Response.CreateResponse()
  • Return HttpResponseMessage created in the controller explicitly with a MediaFormatter

Quick, which do you think is faster?

Here’s what the three different methods look like:

[HttpGet]
public Person HelloWorldJsonTypedResult()
{
    return new Person();
}

[HttpGet]
public HttpResponseMessage HelloWorldJsonCreateResponse()
{
    var response = Request.CreateResponse<Person>(HttpStatusCode.OK, new Person());
    return response;
}

[HttpGet]
public HttpResponseMessage HelloWorldJsonManualResponse()
{
    var response = new HttpResponseMessage(HttpStatusCode.OK);
    response.Content = new ObjectContent<Person>(new Person(),
                    GlobalConfiguration.Configuration.Formatters.JsonFormatter);
    return response;
}

If you look at the performance result of the benchmark test shown earlier you’ll find that the fastest to slowest order is:

  1. Manual Response Creation
  2. Request.CreateResponse()
  3. Typed Result
  4. String Result

It’s also surprising that typed results and in particular string results are quite a bit slower 5% to over 10% actually. I would have thought that string conversion must be one of the easier things to do especially given that there’s no object parsing required.

Self Host – Sloooooow

Another surprising thing is that Self-Host is slow using the same controller used in the Helios example. It’s very nice that you can pretty much reuse just about the exact code from the Helios sample with a self-hosted server – only a minor change in the startup class is required. But performance – at least on this Windows 8.1 desktop machine, is roughly half that of the Helios IIS example. IIS is clearly optimized for HTTP operations, but I expected that HttpSys based self-hosting would be at least close, if not faster than IIS since there’s no overhead at all going directly to the Web API or plain through OWin endpoints. Just raw Http.sys.

There’s been a lot of talk recently of using Self-hosting rather than IIS for efficiency and a leaner platform in general, but that doesn’t necessarily translate into better performance if the host technology is slower. IIS is pretty darn efficient, stable and performant and it doesn’t sound like a good idea to me to abandon the tried and tested IIS platform, unless there is a very good reason to do so (ie you need to embed HTTP functionality into a non-Web application).

Summary

These tests aren’t meant to be definitive end-all – I basically have this suite set up so I can quickly gauge performance of the various technologies at a glance. The code for all of this lives on GitHub, so you can install and test this yourself and add your own custom scenario tests to these existing tests as needed. I’ve updated the project with the tests described here and also tweaked some of the batch files and test result parser.

Posted in ASP.NET  IIS  OWIN  

The Voices of Reason


 

Joe Brinkman
December 10, 2013

# re: Re-examining ASP.NET and Helios Performance Tests

Rick - Is self-host multi-threaded? Without that, I suspect that self-hosting will be limited to whatever throughput it can achieve on a single core. This was one of the main limitations for node.js and a major feature in the IISNode implementation.

Doug Dodge
December 11, 2013

# re: Re-examining ASP.NET and Helios Performance Tests

Rick, I've been fooling with Helios a little. Had to add a reference in addition to what you had posted. Suspect my configuration and I REFUSE to rebuild my machine. *G* Worked like a charm when I got things set right.

First of all, thanks for posting this. I always feel like you give a full meal, so to speak, and I am grateful for that.

Secondly, when I have time, I intend to explor this a litle more. Looking forward to what is finally delivered. I can see some useful applications coming from this technology. Speed still is king. *s*

Rick Strahl
December 13, 2013

# re: Re-examining ASP.NET and Helios Performance Tests

@Joe - yes self-host is multi-threaded. HttpSys provides threaded events - there's no way you'd get 5000+ requests a second on a single thread me thinks :-)

Y#
December 15, 2013

# re: Re-examining ASP.NET and Helios Performance Tests

Ah, and yet another great article-to-the-point to read from you Rick! And you bet about the anti virus thing... I've known for a while that I have to disable it (after cutting my box from any network) every time I want to profile closely some piece of code I don't want to be disturbed (by the %CPU fluctuation) Oh yes, confirmed!

Speaking of which (performances)... just thought I could share, hopefully not totally irrelevant:

https://github.com/ysharplanguage/FastJsonParser

;-)

Cheers,

Shaun
December 19, 2013

# re: Re-examining ASP.NET and Helios Performance Tests

Amazing the performance difference Windows Defender service causes. I'm running Windows 8.1 on an i7-3770, I have a simple Helios app that outputs "Hello world", with Defender running I get 4,700 reqs/sec, with Defender stopped I get 12,200 reqs/sec! Insane. A lot of the time when I restarted Defender, it'd hog 100% for a long time, or completely hang like that. Weird.

That aside, I notice when I run a long benchmark, the IIS worker process is still only using 28% or so CPU, no matter how many concurrent requests. Would this be an IIS configuration issue?

Rick Strahl
December 20, 2013

# re: Re-examining ASP.NET and Helios Performance Tests

@Shaun - yes defender has a huge impact for me too, but not quite that much. Defender must be looking at each network request hitting IIS and so effectively slowing down the pipe.

Re: IIS Perf. I think what you're seeing is some internal throttling of IIS on workstation OS. There are no longer connection limits as there were in XP and Vista, but internally IIS still throttles. I think if you tried this on a server OS you'd be able to saturate the CPU if you ran enough instances.

West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2024