Rick Strahl's Web Log

Wind, waves, code and everything in between...
ASP.NET • C# • HTML5 • JavaScript • AngularJs
Contact   •   Articles   •   Products   •   Support   •   Search
Ad-free experience sponsored by:
ASPOSE - the market leader of .NET and Java APIs for file formats – natively work with DOCX, XLSX, PPT, PDF, images and more

Azure VM Blues: Fighting a losing Performance Battle


On this page:

I’ve been looking at Azure for a long time, hearing all the glowing reports about how well integrated the platform is and how easy it is to move things get up and running. I’ve been contemplating moving to some sort of virtual host from my current physical server setup as I can certainly see the appeal of having a managed platform that can be remotely administered, re-configured on the fly if needed and provide an easy way to backup and duplicate the environment since it’s all based on repeatable VM images. The promise is great.

This post is a bit of a rant and chronicles some of the issues I’ve run into with Azure VMs. It’s not meant to bash, but rather I’m trying to document what I experienced and to see if maybe – just as is often the case – I hit a weird edge case. I’m also interested in what others see with Azure in terms of performance and stability and consistency.

For me personally I find myself in an odd position where I have around 20 web sites that I currently host on my own physical box that I run at my ISPs site. Most of these are personal sites like this blog, a message board, my online store and a host of other internal business services. I also host a couple of small sites for some of the clubs/associations  I belong, plus a small commercial site for my girlfriend.  In short the hosting I’m talking about is essentially personal hosting that is not paid for out of a commercial budget, but personal money I have to lay out every  month, which changes the perspective of how I’m approaching Azure here.

Where I’m at now

My old server is aging – it’s a Server 2008 machine that’s going on 6 years old. The hardware is doing just fine for the apps that I’m hosting – two of which generate a fair bit of traffic in excess of a million ASP.NET hits month each. Still the hardware is barely even taxed at all by this. However, I’d like to upgrade the server to a more recent version of IIS that supports WebSockets and also the forthcoming ASP.NET vNext which won’t run on Windows Server 2008 (or at least it doesn’t currently) so I’m due for a complete repave.

My current choices are moving everything off the box to a virtual machine – Azure or otherwise (and after moving a few sites to a new VM I’m leaning towards the latter by far), or installing a newer version of Windows on my aging server. To make the decision even more difficult, my hosting arrangement with my ISP is such that I have a very generous grandfathered rate that makes hosting there very cheap – pricing that Azure can’t even begin to touch. I don’t expect the same rate for anything I move to but I’d like to be at least in the ball park, which as it turns out is not easy with Azure – comparatively speaking it’s very expensive when trying to match the existing hardware I’m running on.

Experiment: Move my Blog to Azure

My blog is one of the apps running on my server and it’s the busiest of my personal sites running on the old server. The blog is fairly isolated app without dependencies, but it does have a boatload of data in it between blog entries and comments over the years (as most of you know my posts tend to be looooong).

In any case, since I have an MSDN subscription and Azure credits I figured I’d take the opportunity to move some of my sites off to a new VM on Azure and experiment to see what kind of performance I really need for a VM.  I’ve started this process on several occasions in the past and each time I’ve backed off because performance was either not anywhere near what I was hoping for or the price immediately shot up to a level that I just can’t justify. Since then prices have come down some, but still not quite in the range I can really deal with for the 20 or so personal sites I’m running on my box.

To start I moved my blog over onto the VM as my blog is by far the busiest site that I run and it would be a good indicator for CPU and bandwidth usage and how that would fit into the Azure hosting plans.

Installation and Deploying

I decided to create a virtual machine with the following setup:

  • A2 Instance (2 cpu 3.5gig) – switched to D2 later
  • Second data vhd in Blob Storage

The process of creating new VMs and getting them up and running is very easy and the process takes about 10-15 minutes until you have a live instance to RDP into. That part’s really nice and it works well.

Do it right: Don’t use Quick Create

However, it wasn’t all smooth sailing at first, because I took the easy route at first.  I had made the mistake of creating the VM with the Quick Create option and while that works just fine it creates a mess of dynamically named components. There’s a whole slew of them Cloud Service, Vm, Storage Account, Storage Container, Disks and VHDs.  Additionally I went through the options too quick and ended up creating my site in the Western Europe region instead of West US.

Deleting VMs and associated resources is a pain

End result I had to kill everything and start over. Now that should be an easy task, but it turned out to be quite a pain as you have to delete all the component pieces individually and in the right order. Otherwise components end up locked and you can’t delete the parent components. I also am not fond of all these nested containers of container of containers. I’m sure that’s needed for the implementation, but why do we have to see this information in the portal’s UI? All I should have to do is delete the VMs and delete the VHDs.

To make things worse files ended up being locked. I was able to delete the VMs but that left behind the VHD disks and storage accounts. Unfortunately these were locked and for the better part of an hour I was unable to delete them.

Manually install everything

After I finally managed to completely clean out my Azure account to start fresh, I re-did the install properly by individually creating each of the components. Storage account first (for the second disk to attach) then manually create the Windows VM and go through the long form Wizard to name everything explicitly. It took a little more time, but now at least everything is reasonably organized. Much nicer and frankly the way it should be – the Quick Create Wizard should not even be an option!

Up and Running

Once I was up and running I logged on with RDP tried to install a handful of apps required on the server. The main piece was activating the Web Server role and installing IIS, and SQL Server locally (yes I know this should be off the machine, but I don’t have big db access so the server is and will stay local on the box).

Slow Desktop

As mentioned I started with a A2 instance which is 2 cores and 3.5gig. Now typically I would peg that at the low end of the hardware spectrum for a Windows Server box, but I figured this would be a good place to start for hosting. I quickly found out that running an A2 instance even to install a bunch of applications from the command line was unbearably slow. I used Server Manager to enable IIS which took seemingly forever (at least 20 minutes) and then used Chocolatey to install a bunch of my utilities and dev tools first including Chrome. It took insanely long – at one point I went to the store to get lunch and came back and still found SQL Server installing when I got back an hour later.

The machine was unbearably slow using RDP. The UI would constantly hang up and apps would show “Not Responding” whitescreening even the Windows desktop. I’d get a burst of activity when everything works smooth, followed by complete lockups for several minutes where I could move the mouse around the desktop but nothing is responding.

For example opening Chrome (after initial install and several times of opening) took nearly a minute. Trying to bring up TaskMan failed altogether – or so I thought until after about 5 minutes 5 TaskMan Windows popped up because I had clicked multiple times thinking my click hadn’t worked.

nGen?

When I did manage to get Resource Monitor to pop up I took a look to see what’s going on that’s making the virtual machine so crazy slow and to my surprise I noticed that CPU usage was low (under 20%). But I did find disk activity maxed out. Taking a closer look I noticed that the mscorvw.exe was hogging disk access like crazy. This the ngen compilation service that runs in the background after .NET framework installs or updates and apparently it was seriously hogging machine resource to the point that the machine was slowing to a crawl.

I figured I’d call it a night and let ngen finish after explicitly forcing it to complete all compilation with:

ngen.exe executequeueditems

Next morning when I came back I restarted the VM and when it finally came back performance was at least usable, if still very slow in RDP. No white screen lockups but still massive lags between any click operation and painfully slow execution of anything that is actually running on the machine. I even turned off server optimization and put the machine into ‘desktop’ mode, which made next to no noticeable difference.

To give you a point of reference how slow this stuff is: It took SQL Server Management Studio over 2 minutes to come up for the first time after a reboot.

By comparison my 6 year old 4 core physical server feels like you’re there – on a slightly older desktop machine that it is.  The RDP connection feels almost live with only the slightest bit of lag and applications run with appropriate speed. It doesn’t compare to my current gen laptop, but it certainly matches the perf I would expect from a 6 year old machine. On the Azure machine – even though the CPU seems to indicate no heavy load – everything is just sludging along.

And what’s really strange is that even though the machine feels nearly unresponsive, running Resource Monitor or Taskmans perf tab, I see barely any CPU activity. The CPU hovers in 15-30% in spikes but nothing more than that. Sure seems odd, and looks like a case of a way over-provisioned machine.

Trying other Azure Instance Types

At this point I decided that maybe an A2 instance is simply not enough hardware. A2 is 2 core with 3.5 gig of memory which to me seems pretty low, but looking at resources used on the machine at this point, the A2 doesn’t look like it should be having problems. Committed memory is under 2gig, and CPU usage is generally in the 15-30% range.

I tried loading up an A3 instance anyway to see if the extra memory or CPU would help. A3 has 4 cores and 7gig which is more comparable  to my physical box. It seems odd to want to add more CPU when the CPU doesn’t seem to be taxed on the A2 setup. However, as expected I saw no performance improvement whatsoever with that instance type. The CPU usage was the same (but spread over 4 processors), but actual performance of the desktop was absolutely no better than before.

On the plus site it’s extremely cool to be able to take an existing application and throw it onto a new hardware setup without having to change your Windows config – that’s pretty awesome and one of the big selling points of a Cloud platform.

I did get a new IP address so if the slow perf was an issue of ‘noisy neighbor’ (one or more busy sites on the same physical hardware), then this problem now carried across multiple machines.

D2 Instance – better but still not good

Next I tried the new D2 instance. D2 is 2 cores – 7gb, but running on SSD drives. And that finally DID make a decent difference. Performance improved drastically with a D2 instance and the machine finally felt like an approximation of what it should feel like given the hardware that is running it.

So it seems to point at the problem being the abysmal default drive performance of the A2 and A3 instances which seems insane. While I can understand somewhat slower perf with a non-SSD drive, it doesn’t quite account for the HUGE difference in things like launching apps and plain interactive behavior of the desktop which are not typically disk bound especially on repeated loads. There’s gotta be more to it than just the SSD drives.

I ended up switching back and forth a few times between different instance types – getting new IP addresses most times and presumably jumping around different machines to mitigate the noisy neighbor problem, but every time I went back to A2 and A3 instances I saw the same unbearably slow performance. It never got any better. And switching back to D2 I saw improved performance but still not anywhere even remotely close to the performance I’m used from my physical box.

I’ll have more on performance when I talk about some Web app testing later on.

D2 for Installation

For the time being – while configuring the machine - I left the machine running on D2 so I could actually get shit done configuring the machine. On A2 it was just too painful to even run the command line where the CLI typing couldn’t keep up! That tells you how bad it really is!

To be fair this ability to switch instance types is one of the very cool and truly useful features of a cloud platform – you can temporarily set up a machine to run a faster (or slower) instance and then kick it back to your default setup which can be very, very useful and lets you get stuff done.

With VMs you have to be careful with live installations though – reconfiguring a VM means re-provisioning and that takes some time. I’d say for a VM to start back up in a new mode and be reasonably functional takes about 10-15 minutes and it will be pretty slow for a while after that after all the components on the box update, indexing and ngen runs again…

I also find it odd that the Azure portal shows VMs as ‘running’ after a few minutes, but it takes another 5 minutes more  before the VM actually becomes accessible to the Web site or RDP. It takes closer to 10 minutes to reprovision. Machine reboots also seem to take quite a long time… I suppose this all comes back to the slow processor performance.

Installing My Web Blog app

Despite all of this, I decided to at least get a Web app up and running to see how an actual running app would fare. After all, background services tend to perform better and maybe, just maybe these Azure instances are optimized for services.

The WebLog is a pretty simple app, but it pulls all data from a local SQL Server database. The local Sql DB file is installed on the C drive since that’s the SSD drive at least on the D2 instance.

So I installed all needed IIS components (next time I set up a Chocolatey script for this!) and configured the WebLog Web site. Created a Web site and then published the WebLog from my dev machine to the server.  I backed up the database on my physical server and downloaded it from new machine and restored the backup into my VM’s Sql Server.

The WebLog app came up first try which was good to see… I had to tweak a few disk permissions to allow local file writes for publishing posts and images, add the SSL cert for some of the admin tasks, but otherwise this process was pretty painless. A nice reality check… A software is easy compared to the Admin crap :-)

Stress Testing

At this point I had dialed the VM back to the A2 instance and let it run for a while, explicitly ngen’d and put the server back into Background Service mode. I also let the server run for a bit under a secondary domain (http://weblog2.west-wind.com) which I would hit occasionally to keep the server up and running and give the OS some time to stabilize with updates and housekeeping compilation (indexing, ngen etc.).

Initially I tried running external stress tests over the open Internet but quickly found out that Azure has some Denial of Service filters in front of their servers that make external load testing tricky. The results were very slow and erratic as heck, even though the live site seemed responsive in the browser. I figured that Azure is throttling traffic so remote testing is not really an option or even a good guideline for testing machine performance anyway. So don’t bother externally testing load of an Azure VM or site.

So afterwards I RDP’d back in and installed and fired up WebSurge locally to load up the Web app to see what kind of performance I could get out of it. The nice thing about this is that it’s a simple test that easy to repeat on both machines running exactly the same Web application.

This was a total disaster on the A2 machine. Tests were very slow responding to requests and almost immediately timing out when sending continuous load:

A2 Instance – local machine:

A2SlowWebLog

9 requests a second is pretty sad for performance, but even worse there were 10 failed requests that actually timed out. Also check the request times that are over 2 seconds in the best case and close to 10 secs in the worst case under load.

By comparison and for reference, here are the results from my live server running the very same setup:

West Wind 6 year old Physical Server – 4 core low end Xeon

LiveServerMachine

That’s over 200 req/sec for these very content heavy pages which is more than 20x higher load than the A2 instance!

Next I switched the machine back in D2 to see how that would stack up.

D2 Instance – local machine:

d2Windows

You can see here I’m getting 54 req/sec which is a lot better but still not anywhere close to the performance of the physical box. Test here also ranged wildly from 55+ req/sec down to less than 10 req/sec – often in short succession to each other. I can’t make sense of these tests – there’s no consistency at all here.

Even in the high range when the tests are running well, this is still nearly 5x less than what I get consistently on my server.

After quite a bit of back and forth testing on both A2 and D2 instances I’m also seeing the same performance flux across all machines. I logged on again today as I’m writing this and trying the tests again, and now I’m seeing barely 10 req/sec for the D2 machine and it too was showing bunch of timeout errors. Then I log back in a couple hours later and try again and I’m back up in the low 50’s as before.

Keep in mind this machine has currently no traffic other than my testing accesses. I see 60-70% performance swings which is totally unacceptable… By contrast my live server is pretty steady in tests which vary by less than 3% and that is with actual traffic on the various other sites that are running on that server. IOW, that’s what I call predictable performance – Azure, not so much.

Can it really be this bad?

So, at this point I’m thinking I have to be doing something really wrong to see performance that’s this abysmal compared to relatively low end  physical hardware and for the performance to be so incredibly erratic. I’ve pretty much ruled out the noisy neighbor problem because of all the jumping around of IP addresses that has been happening.

For the Web stress testing I initially  pushed external requests in from my local machine here, but I quickly assumed that Azure’s front end that sits in front of the VM might be doing throttling on the network end to prevent denial of service attacks. That would have explained the erratic behavior of requests. But after also testing locally on machine I see the same behavior running on the machine against localhost.

What’s interesting in local testing I can see the request load getting slower and slower. I start out with maybe a hundred requests that take 100ms or less and then it bumps and slowly the request times are creeping up and up and up until they reach the 15 second timeout. That certainly seems like some sort of throttling is kicking in, but I can’t see anything that would do this locally. Anybody have any thoughts of what locally could do this? IIS has Dynamic IP Restriction module which can detect and restrict addresses based on traffic loads, but I double checked and the features are not enabled on this box.

I can’t image that Azure could be sitting between an HTTP client and local connection going against localhost, or could it?

While running tests though I can see the external site when accessed from my machine also start to lag – so that at least seems to corrobarate that I’m not imagining things here – when busy I see 10sec wait times for external requests too. So either the site is basically bogging down due to limited hardware resources or there’s some throttling which affects everything.

And there there’s other indication of slowness via the terrible RDP performance. Even on the D2 instance the UI is laggy as hell, and most applications take forever to launch. Also when hitting some of the longer running Admin links on my WebLog site (cleaning up entries, shrinking Db etc.) these requests that take 10-15 seconds on my live server are timing out the 3 minute request timeout of the server on every hit. So even there performance is seriously lacking.

Expensive

The other issue I see here is competitive pricing. Running even the lowly A2 instance full time is $133 a month – which is roughly the same that I pay today for my physical box hosting. But going to D2 – which at this point seems like the absolute minimum that I could see my self using, almost doubles that amount.

Granted that if you’re running a business that’s generating revenue that’s probably reasonable, but if you start comparing this type of an environment to traditional co-location or machine hosting which is half of that cost and provides significantly better horsepower, it’s really hard to get excited about virtualization.

Sure, there are lots of services that you can take advantage of in Azure and the remote administration and quick standup of resources is pretty awesome – no doubt about it. It’s a different kind of business model altogether. But in the end performance should still matter a lot and the bang for the buck seems pretty low compared to traditional solutions when it comes to performance. The trend here is going the wrong direction.

I realize I kind of have an oddball set up here in that I have many sites I’m running which is why I think I really need a VM to make it cost efficient versus running Azure WebSites and SQL Azure. When I tried to price out running just my Blog on Azure WebSites plus SQL Azure it amounted to almost $100 for the single site (WebSite + SQL Azure). Again the ROI there is not anywhere near the physical box when you figure in 20 sites. Then account for the mish-mash of server software (SQL Server, MongoDb, IIS some old FoxPro stuff) you realize that anything but a VM is not really an option. I’ve also found that even when a hosted solution could make sense I’ve had issues with some piece of .NET not being supported (System.Drawing specifically – I have a few graphics/chart libs I use that just won’t run) so I have several apps that couldn’t run that way anyway.

If in the end I only need virtual machine hosting and no other services, Azure is also very very pricey compared to some other hosters. Specifically I recently looked at Vultr which is hosting a 4 core 7 gig Windows VM for $84/month on SSDs. I’ve signed up but haven’t tried them yet. To be honest I think ~$100 for this type of setup seems to me at least an acceptable rate. On Azure there’s no exact match for this setup but it sits between a D2 ($257/mth) and D3 ($509/mth) instance which seems like a lot especially given the mediocre performance I’ve seen so far. Reviews I’ve seen are favorable for performance – we shall see.

Is this the Trend?

I’ve been dealing with Virtual Machines of various customers in the last few years, and in general I’ve gotten the feeling that VM hosting is a serious step backwards for performance. Most hosted solutions either run on low end hardware that is drastically slower than processors were even a few years ago as well as over-provisioned machines that are yo-yoing all over the place when it comes to performance.

When you get 2 cores assigned – what are you actually getting? 2 cores on a 4 core machine that has also been provisioned for 10 other customers? Yeah that’s going to suck performance wise.

Who measures what the performance of the cores we’re getting in a VM actually is? There’s no easy way to compare performance because it’s totally variable and well ‘virtual’. Most providers tell you a number of cores they provide but not what type of chip and actual speed and how many virtual processors are allocated to each machine/core you’re getting. It could be 4 Atom processors for all we know :-).

Using VM software you can spin up any number of cores even on a single core system so how can we even be sure that we’re getting ‘2 cores’. Does that even have any meaningful value?

The Azure experience I just related seems to corroborate that exact scenario. So I’m skeptical even though I would absolutely love to move my machine over to a VM.

What about you?

So, what are your experiences with Azure VMs? Am I just being overly negative, or maybe better – am I just missing something that’s not configured right? How about other VM hosting for Windows? What have you tried – what works and is reliable?

**Update 1:

Once the site and SQL Db were imported from the remote site, running the first batch of load tests resulted in this:

Vultr 4 core, 7gig:

Vultr[4]

This is getting pretty close to what I get on my physical server box. Further – performance was consistent. Looking at CPU stats I can also see the CPU getting loaded up the way that it should on a stress test like this, which again makes me think that Azure has some sort of throttling going that is somehow detecting the load testing and choking it off.

Anyway – I find it pretty interesting that a small provider can provide a much faster experience for a fraction of the cost of Azure. I would expect Azure to have a premium for all the extra services and convenience, but – I would also expect Azure to lead in or at least be competitive in performance. From what I see here – I can’t say that it is.

Update 2:

After a number of comments and messages that I should not be seeing A2 performance that’s quite this bad I set up another A2 instance – this time using all defaults rather than pre-creating the second drive. And it turns out performance is quite a bit better on that setup (or possibly better luck with noisy neighbors – I dunno). With this setup the UI is reasonably responsive and the load test performance is also looking much better.

A2 instance

A2Improved

This is a LOT better than my previous install and in fact better than the perf for the d2 instance I had before so this is a huge improvement. But in light of having installed a machine on Vultr which is still 3x faster and cheaper it’s hard to get excited about it at this time…

This still begs the question why performance of the other A2 instance is so absolutely terrible. Although I pre-created the storage container, I didn’t precreate the actual boot drive – only the second D: drive so I still can’t account for this terrible performance.

After some more checking with the new A2 instance I can also confirm that the performance seems a lot more steady on this one with the expected < 5% varience in tests at various times.

One upside to all of this is I’m getting together a pretty good script to configure a machine quickly. The only manual install and huge time sink in the whole process is the freaking SQL Server install.

Posted in Azure  

The Voices of Reason


 

chris kinsman
February 01, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Not sure I follow azure website pricing. Small basic instance is only $55 / mo and allows multiple sites. I suspect performance would also be better for your scenario but would be very interested to know your results..,

Rick Strahl
February 01, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

@Chris - Uhm, yeah sure that works for some simple scenario or testing. But if you actually have a bunch of stuff you need to run that's going to be a problem very quick. Eventually you're going to have to go to the next level and probably the next one after that which puts you in the same scenario I describe above with the single D2 VM as a minimum.

If you price out a bunch of sites individually (with data as a service) it'll get more expensive than a single VM very quickly.

Marko Lahma
February 01, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

If you have anything I/O intensive like databases you should try striping disks and having separate disks for data/indexes/logs. You only pay for used bytes so it's kinda cool that striping is "free". Downside is that VM type dictates the allowes disk count. But something to play with maybe, even though your load does not seem to be bound by I/O.

And as Chris noted, single website can host multiple sites with same price.

Jeremy Blosser
February 01, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Definitely sounds like you didn't set up an affinity group. The result being the high disk access times since your VM on the A2 instance would be accessing a VHD "further away" than if it was in the same affinity group (which gets your resources "closer" together within the same region).

It is a pretty common mistake and one that "quick create" handles for you, but since you opted to create your own, it is probable you missed that step.

Rick Strahl
February 02, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

@Marko - on the D2 disk I would think SSD should avoid any data bottle necks. Data usage is not very heavy - Db access yes but pretty basic stuff (probably all cached during stress test).

@Jeremy - hmmm that sounds quite plausible actually, but the blob container is definitely in the same data center (both the VM and Storage account are in West US). This also still wouldn't account for the slowness of local apps all installed on the C: drive - like launching SQL Server. Or are you talking about something different?

Nuno Agapito
February 02, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Definitely something was miss configured or there was some problem on the West US servers...

I have 5 A2 standard azure servers running on the West Europe servers and I dont have any of the problems you described. Yes, I might have less load on the servers but even the RDP and UI issues you described are something that I never experienced.

One of the servers is a backup development ambient server, just in case I need to code a fix and dont have my PC near, running SQL server and VS 2013 and the only problem I notice is the normal slight delay while typing due to the latency.

Rick Strahl
February 02, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

@Nuno - Not sure what you might be doing differently. This isn't the first time I've seen this dreadful performance - as I mentioned I've played with this stuff on and off for over 2 years and always turned away because the performance was so abysmal.

To be sure I'm going to install another A2 instance using all default settings to see how that goes.

For the tests I'm running I'm expecting to performance ~100 req/sec to even remotely consider using it.

Steve
February 02, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

The post about the D machines only mentions that the temp D drive is on a SSD drive, not the whole C drive.

http://azure.microsoft.com/blog/2014/09/22/new-d-series-virtual-machine-sizes/

RyanCEI
February 02, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Rick when I was going through a very similar migration I found that SQL Server does not play well on the smaller VMs. I now only host my IIS sites and some basic console apps on the VM and use Azure SQL for all db work. I realize that it's note feasible to migrate your legacy sites to Azure SQL but you may want to consider a split configuration for your next greenfield project.

Rogelio Martinez
February 02, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Is there any reason you didn't go the azure website / SQL azure route instead of a full fledged VM ? It would better fit your use case in my opinion.

PilotBob
February 02, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Isn't running Web Sites always going to be cheaper than a VM instance for hosting? With a standard Site you can run multiple sites.

A SM Standard 1 VM Web Site is $74.40 a mo. 1.6 GHz CPU, 1.75 GB RAM

A single A2 VM is $113/mo.

So, you could run two standard Web Site VM instances for $148 a /mo (small instances) I highy expect that size would easily support your blog and a few small sites. Of course, if you go this way you would need SQL Azure as well.

This all said, I'm sure there are probably cheaper hosting options for personal or very small business use.

Jeff Putz
February 02, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

I've definitely hit some issues, but moving to Azure Websites has saved me a ton of money. I've got about a dozen sites and two S0 SQL databases. Running on a small instance I'm experiencing a little bit of memory pressure, but it's usually not an issue. I'm happy to not maintain a server anymore. I'm seeing consistent SQL access for most stuff below 20ms, and page rendering overall under 60ms (my junk is all MVC-based). I have a handful of little things running as Webjobs, some queues, a Redis cache, and I'm experimenting with search. Overall, I'm pretty happy with it. I wrote about it here:

https://weblogs.asp.net/jeff/i-moved-my-web-sites-to-azure-you-won-t-believe-what-happened-next

I'm not sure if you're in a position to ditch VM's, but I dig the PaaS stuff.

Rick Strahl
February 02, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Thanks for all the comments and another nudge to retry the A2 instance with a fresh install which has improved things a bit, although not anywhere near the perf of the Vultr VM which is cheaper and provides perf nearly on par with my old physical machine (or 4x better than the A2 box).

@Bob, I think the way WebSites work is you essentially pay for the WebSite (which ends up standard VM) and then you can host multiple Web sites in that VM. A little more pricey up front, but cheaper then for as many sites as you can cram into that vm.

For me though I have 20 sites and some of them I know will not work as Azure Web sites. Sites also don't provide for SQL which you have to pay on top of that base fee. It all adds up.

I agree with others here - for pure greenfield WebSites make good sense but for legacy you have a lot of things that can go wrong (non-supported features, system level access for admin stuff etc.). So I'm pretty sure that a VM is the only way to go.

The other issues are data access. I use SQL for most of my older stuff and MongoDb for a bunch of newer stuff and some of the Mongo app have a large chunk of data in them. Moving those off to a SaaS setup would be pricey.

I'm going to set up the blog as WebSite just to see how that goes - not quite sure how to get everything over to the site since there are a lot of content folders for the blog (all the images over the year) - there's about a gig's worth of disk space and about 3/4 gig of data in the db.

Matt
February 02, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

While my sites work and fit well in the PaaS/Azure web site model, I've had the need to spin up Azure VMs in the past and I too have seen alarming performance issues. Just basic stuff like interacting with the desktop or console over RDP can be maddeningly unresponsive.

Responses like "well, why aren't you just using Azure web sites?" or "you need to upgrade to a higher perf tier" aren't helpful. I'm curious to see if you are able to arrive at a definitive reason for the issues, as I'm working on a web application that is going to eventually outgrow the web site model and is going to need a VM.

Rick Strahl
February 02, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

@Matt - I just went through the process of creating 2 other A2 instances and it is indeed maddening to see how widely the performance varies. The first instance i created last night ran pretty fast (I posted an update on that).

But I had used all the defaults. I then raized everything I had previously installed and create just a new storage container to house my windows drive, then created a new A2 VM and let it create the Windows drive in my storage container. Again the perf is terrible on that one.

Not sure if that's definite but it's possible that the external storage is slower or further away from the VM that it's causing these problems - I'm not sure but this is frustrating as heck.

Bastian Schmidt
February 03, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

You can also automate the installation of SQL-Server.

https://technet.microsoft.com/en-us/library/dd239405(v=sql.110).aspx

I built a private chocolatey package for that at work which downloads the iso (repackaged as zip), starts the installation using the ConfigurationFile.ini and then adding the required firewall rule (netsh advfirewall firewall add rule name="SQL Server" dir=in action=allow protocol=TCP localport=1433).

Feel free to ask me on twitter if you need some help.

Mike
February 03, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Take a look at the limitations of IOPS on your azure machines.
On smaller instances you get 300 IOPS per disk. More expensive instances get a 500 IOPS limit per disk.

Especially if you have SQL DBs on a VM, you have to create as many disks as possible and then use them as a RAID-0 stripe set to store your DBs on.

John
February 03, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Rick, I share your frustration with performance problems. I was looking into moving my entire company (hosting of 150+ asp.net sites using a custom CMS) out of a local datacenter and onto Azure. First I looked at Azure websites, but my sites use Sql CE and the performance of Sql CE under Azure websites was terrible. I posted in support forums looking for answers but could not get a reasonable explanation.

After the issues with Azure websites I decided to try VMs. I actually had pretty decent performance with these (much less latency than what you described), though it tended to be inconsistent and I lost access while I was testing during of one of their big outages. After the outage ended I had a benchmarking app that I wrote to test Sql CE vs Sqlite on Azure websites and ran it on the VM, and performance was significantly better in the VM than on Azure websites.

At the end of the day though I gave up because performance was inconsistent, there were major outages while I was testing and I was nervous about static IP reservations. I just do not sense complete reliability when I am using Azure, it seems like there are little bottlenecks and hang-ups all over the place, enough so that I just cannot justify moving production systems over yet. When I read articles like yours it gives me the sense that I was wise to wait. Curious to see if you manage to get things sorted out in any reliable way.

Edward
February 03, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

As a Azure user I can say that I know a few pros and cons.

We use VMs a lot because we needed a lot of servers fast and soon. Azure is great with this. Performance wise its doing a great job except voor lots of IO. MSSQL DB's tents to be slower then when you go dedicated yourself.
If you want 99.999% uptime you need to go with 2 VM servers in one Availabilty set. Otherwise you notice a reboot every 3 months or so (maintenance). This is why you need a mirror when using MSSQL (enterprise will cost you a lot btw).

Thing is: why do you want to go VM? Why not Azure websites? Your ROI calculations seems wrong. You can have more then one websites on one server.
Unless you have very specialized whises and some kind of 3party components which are COM my advice is: Go Azure websites. It is full managed, pretty painless and Visuals studio Publishing wise very nice.

Try to experment with it.

Edward
February 03, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle


Steve
February 03, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Thanks for this post. Again, amazing how years of "simplification" and fall short of the marketing claims.

Max
February 03, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Would filling out the Azure penetration testing whitelist form help with this?
https://security-forms.azure.com/penetration-testing

washu
February 08, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

As someone who has used many of the big cloud providers all I can say is this: don't use Azure. I'm not going to suggest who you should use because it really does not matter, they are almost all better and faster than Azure. Try someone else, anyone else and you will see how bad Azure really is.

Mika Kukkonen
February 09, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

I did a similar study two years ago with Azure North Europe Region.
http://www.flowmarks.com/2013/02/migrating-web-application-to-azure-vm.html

My tests focused on average page load times rather than throughput.
I measured page load times for a single user to see if the best-case performance in Azure was competitive with an existing system.

Results were similar, the physical server (a 2006 low end machine) generally outperformed Azure.

I concluded that because the application made many requests per page, round-tripping between Helsinki and Dublin (2000 km one way) was the main cause of worse load times. There might have been other Azure-related latencies, but if the bottleneck was the database, Azure was actually faster.

Network latency could also explain your problems with RDP.

But overall the Azure performance was so unreliable that in the end we decided to buy new physical servers to replace the old ones. This was a very good investment, because low-end server performance had improved dramatically. A low end server today would be something like a 4-core Xeon, 64GB RAM and a 256GB SSD Raid-1.

At the time, Azure prices were competitive with other VM providers in North Europe, but I haven't checked lately.

Rick Strahl
February 09, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

@washu - it certainly looks that way. I stood up a couple of VMs on Vultr and perf is nearly the same as my physical machine and costs a fraction of what Azure charges for the same setup that doesn't perform half as well. Azure is just not even close to competitive on performance in that comparison. I have to say I did not expect that when I looked at Azure - I figured it would be top of the line experience in terms of performance.

Microsoft better get their shit together on this - great infrastructure, which they certainly have, is not enough to make up for bad performance. After all bad performance means you end up overpaying for resources you otherwise wouldn't need.

Giedrius
February 11, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

It would be really interesting to see how Amazon EC2 performs in this context. Btw, EC2 had CPU throttling on smaller instances - if you was hitting CPU too hard, it would limit CPU performance for some time, could it be, that Azure has something similar?
Because I've tried some smaller cloud providers in the past and I can say that I had terrible experience with them. EC2 support is not brilliant too, but what works, works quite well.

washu
February 15, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

@Giedrius - As per my previous comment I don't want to sound like I am supporting Amazon EC2, other than it is much better than Azure. This is just my experience with using both platforms.

Amazon has two instance types that have CPU throttling, t1 and t2. How t1 instances throttle is not published, but generally the overall performance is low. T2 instances have very specific and documented throttling parameters based on a credit system and you can view your credits in the AWS console. I have found t2 instances to work very well for "bursty" workloads that otherwise are mostly idle. All other instance types in EC2 are not throttled.

Azure has the A0 instance which is comparable to the EC2 t1 instance and is the only type that is throttled on Azure. The A0 is a low performance shared core instance. All the instances that Rick tested are not throttled. The performance is consistent, just slow. The A series instances above A0 have very slow and out of date AMD CPUs, but the bigger issue is that all Azure instances have absolutely abysmal I/O performance. I have personally done tests on Azure where I was getting seconds per IOP, not IOPS. This is why Rick still had performance issues after switching to a D2. The CPU in the D series is ok, but the disk I/O is still terrible. Note, the SSDs in the D series only refers to the local temporary drives. Unless you are specifically using them they make no difference. Even the A series have local mechanical temporary drives that are much faster than non-local drives. Also, bigger instances do give more sequential disk performance on Azure (and EC2) but since IOPS are usually more important it doesn't help much.

In my experience the current generation EC2 instances (M3, C3, C4, R3, I2) have much higher CPU performance than Azure A series and are still a bit faster than the D series. On I/O performance EC2 is night and day faster no matter what the instance type. Again, this is not a recommendation to use Amazon, just a warning to not use Azure. You will find most of the other major providers are closer to Amzaon than Azure in performance.

Rick Strahl
February 16, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

@washu - thanks for the detailed feedback.

As to D2: D2 did improve performance significantly, but I wonder if that's simply due to the Windows swap file sitting on the SSD drive. Sounds like the CPUs are also better.

This is my general complaint with VM based hosting - it's impossible to compare these services - and often even the different levels, because the CPU numbers and Ghz numbers are nearly useless because it depends on the type and vintage of CPU used. As you say Azure's spec look good on paper, but when compared to other providers that appear the same, it lags behind terribly.

Mark Randle
February 17, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Hi Rick

I noticed similar results especially the extra slow response when doing anything through RDP.

I did think it was the server set up. However I did try Azure previously using a standard try it for 30 days offer. When I did this response was good definitely no RDP lag and also comparable to my current Dedicated Server hires.

I do have a theory though - I like you have credit through Visual Studio and this time I am using this. Surely Microsoft wouldn't throttle this back as we are effectively getting free credit would they ???

Azure was in my migration plan to get away from leasing Physical Servers but not so sure now - Maybe should try again with a public (not free) account!

Rick Strahl
February 17, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

@Mark - yeah I'm using my MSDN account to experiment with this stuff. After all that is what it's supposed to be for. I sure hope they're not throttling MSDN accounts - if they are it's a great way to ensure people won't use Azure because the performance is so terrible :-)

It's interesting to hear responses here that mostly seem to concur on the abysmal performance, but a few here and there seem to suggest that performance is just fine.

Just to clarify when I re-installed new VMs more recently I've had better luck with performance and at least the RDP performance is 'usable'. It's better, but overall with load tests even these newer installs have been very slow compared to other providers I've tried with smaller (and much cheaper) server configurations.

Peter Seewald
February 23, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

We would love to use SQL Database (aka SQL Azure) but the feature set isn't there and running a VM with SQL on it is overly expensive for our shop. Having all of our data and infrastructure in Azure would be way easier to manage and work with but we ended up having on premise environment. We tested performance of our VM versus Azure's VM for a similar setup and saw a similar pattern to what you saw with your older physical machine.

Bottom line is that it if you can't move to Azure Websites and SQL Database in Azure than it's not worth moving. The cost/performance of VM's in Azure for smaller companies/individuals is holding quite a bit of migration to the cloud back. At least in the Microsoft world. Bizspark is decent but when the cost is so high, it's hard to justify that kind of money.

Minnesota Steve
February 26, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

You really don't want to use Azure's SQL databases... They've put a lot of effort into these over the past year to make them perform dreadfully slow. A lot of this I wouldn't have a problem with, as much of it is to throttle bad database design and bad queries. Except they don't have good tooling to help you identify your problems. You can't run the SQL performance analyzer. And worse, you can't easily get a copy of your database and bring it down to your local machine to do this kind of analysis. (even if it was comparable)

Hosting your own database on a VM is a preferable option. Your crappy A2 vm experience is easily equivalent to the $900/month P2 level of Azure SQL.

Marty Glynn
February 27, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

We're getting similar results on Dell - which is apparently an Azure implementation.

roger geisert
March 14, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

I has similar issues. Someone here mentioned setting up an affinity group which I did. Now it's quite snappy.

nom-nom
March 25, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

We recently migrated a medium-ish sized system from two dedicated hosted servers at RackSpace over to Azure. We run a data api that serves 3k/6k (off-peak/peak) requests per minute - about 7M requests a day total to about 70,000 unique clients, some international.

We're having a lot of sporadic issues. In fact, as I write this one of the websites in our group has been returning 503's for 25 minutes for no discernible reason.

We're using about dozen D-series cores of cloud service for our main api app. It does a fair bit of image generation using System.Drawing, GDI+, 3rd party native libraries, etc. An A2 cloud service for recurring jobs and the Scheduler feeding to a storage queue to initiate them. An A2 for a few data ingestion apps and FTP, mounts a storage file share that's shared with the recurring job processor and api. Another A2 cloud service for periodic video encoding and image generation. A handful of websites in one pool - our public storefront and a site just running ImageResizer off our blob storage along with some internal tool sites. 13GB Redis cache.

Our main DB is currently a P3 because Azure has some bug where our database was failing over 10+ times a day and our apps would be unable to connect to our DB for 1-2 full minutes at a time, several times a day. We also use a P1 master and P2 active readable secondary DB.

I can't even begin to enumerate all the many little weird issues we experience, but the end result is that we've barely had a single day go by without all our klaxons blazing at least once. Service downtime is not an exceptional event, it's a matter of course.

There's also no effective way to get questions about these events answered, short of perhaps paying the $1000 a month support plan. Currently we submit tickets and get non-answers after 5-7 days.

Randall Tomes
April 19, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Been using azure for over a year now.
I am on my 6th vm and this one is a small windows server 2008 vm.
Using mvc5 with one simple call to a one table sql database. Nothing else. After a few hours with zero traffic, the machine locks up and has to be reset via the azure portal.

I have also tried webroles and had issues with file content (the entire website) being overwritten with old versions of the website from weeks or months prior. Basically like some auto rollback feature is broken and acts on its own. It consistently has done this 10+ times until I gave up using web roles.

Their support is clueless and they don't even understand azure themselves. Its like the blind leading the blind.

Are there any other more serious, "legitimate" and affordable cloud computing companies that I can use to host .net applications?

Patrick
April 21, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

I am with you 100% on Azure performance issues.

I had a 2008R2 VM at GoDaddy for years, but they will not move to 2012.

I am now have a 2012R2 www.webline-services.com. The uptime has been great and the performance too. I know this sounds like a plug, but it is not. When I moved from GoDaddy I went to a few other hosts and it was a nightmare. It is on my blog.

Ramesh
May 06, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

We had been building our website in an ASP.NET based CMS system in A2 Azure VM's for past 3 months in US East region. The performance of Azure A2 instances had been terrible throughout. We made a few copies of the VM's by creating images during development and each one was terrible in performance with their AMD Opetron virtualized CPU's.

Finally, after the development was complete, we deployed the code & DB in local datacenter on Xeon processors in VMWare VM. There was a day and night difference in performance and the local server performance was order of magnitudes better than the Azure VM (which usually had plenty of CPU and Memory free in task manager).

Based on our experience, I wouldn't use Azure A2 VM's even for running small load web servers in production.

strandedpirate
June 05, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Thanks for this info. We started with an Azure S2 Standard Website to host just a few web applications and the performance is horrible. Averages 2+ second page load times despite implementing every conceivable caching scheme and transport size reduction imaginable and we have next to no traffic. I couldn't imagine how bad it would be if we actually took off one day and got some real traffic. We considered going for a VM to see if performance would improve but given your experience it looks to be pointless.

It really comes down to one thing, because I've lived it first hand, some no name director at MS who's mantra is lowering CAPEX and increasing profits. The end result is every machine is over provisioned and everyone pays the price, literally. This happened internally at a previous employer, DELL. They wanted to reduce CAPEX and moved to virtualize every machine possible and then over provisioned everything. Literally 25-35 VM's running on machines that could at most handle 16. Net result is waiting 5+ minutes from login screen to the desktop. An hour+ to install .Net 3.5 and on and on... When your OPEX is fixed, because they are salary, then who cares right? Wrong, burn out/turn over bonanza. Took two years and 400+ engineers screaming to fix the "policy".

But this is MS and its "for profit" so it won't get fixed and they want it that way. Time for everyone to "scale" as MS calls it (ahem, pay twice as much) or elucidate one's self and find another host. We'll probably purchase a machine and co-locate it a local data-center in the near future. Azure is ridiculous. All the bells and whistles moving at a snail's pace.. no thanks.

P.S> you can't even attach a performance tuner to an Azure database. They want you in the dark, because without knowledge you can't make an informed choice and let rumors and confusion muddy the way forward.

Imranhere2
June 14, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Depressing! What's wrong - Hyper-V or just lots of Hype and bugger all behind it? For SQL I/O is king and its hard to believe they cannot get it right!

EZE
July 03, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

We're trying to run a VEEAM server in Azure as a target. We wanted our backups in Azure so that we could restore them in a DR scenario, or maybe restore for DEV testing. However, even the VEEAM server is horribly slow (A3 standard). Same type of delays as described. Veeam uses local Disk for a simple SQL install, and WAN Accelerator cache. Striped 4TB of disk for backups themselves, but simple file transfers within the VM are lucky to get over 10MB/s. GUI slowness is blatantly apparent. I too thought about doing an affinity group, but Microsoft is strongly recommending against them now, and not giving the ability to create them in the new portal. I really wanted Azure to be part of our future infrastructure (VM's, services, Hyper-v Replica), but I am incredibly disappointed. Having to rethink our entire DR and infrastructure strategy...

Chris Madden
July 21, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

I also learned that network xfer is also limited by virtual machine size and it is different for send/recv. For example for a 4 vCPU A3 instance you can send around 45MB/s, while that same VM can receive 90MB/s. If you increase to a 8 vCPU A4 instance you can push about 200MB/s send or recv, or about 175MB/s simultaneously.

Michael
August 18, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

You really should try transip.eu. I'm using this hoster for more than 2 years. You get 150gig ssd, 4 gig ram , 2 cpu for 25€/month. Nobrainer from where I'm standing :).

Damir Colak
August 24, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

I had exactly the same experience as OP did. I gave up on Azure.

Windows 10, Azure, Microsoft is dead.

Adam
August 28, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Thank you for this post! I've been looking all over for other opinions on Azure because I am definitely getting frustrated. I am in the same boat right now as far as having a physical server and trying to move to Azure VMs. It's definitely tough to see what kind of hardware specs I will get with Azure compared to what I currently have on my physical server, but I really need to move to a hosted solution. I am looking to move around 10 sites to an Azure VM while also running a local SQL Server instance and right now I am on the Standard A0 instance to keep the monthly cost down, but I'm not sure it is going to be enough with 0.25 Cores & 0.75 GB of memory. I have also been looking at AWS and it seems that the RDP isn't as slow but I'm still going to get minimal resources unless I want to fork out a bunch of money (personal budget here as well). I really need to find something that provides decent resources and won't suck my wallet dry, and I don't need all the bells and whistles as far as the admin side goes.

Marc
September 22, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Thanks for this post.
This is the same what I have experienced. I tested it last year and this year again.
I don't have large databases, lots of websites or traffic. I just want to host my website with a simple wish: FAST! But this is not possible with azure. At least not without spending hundreds of dollars per month.

Currently I am hosting a dedicated server at hetzner.de in Germany for around 50$. This is working very well and absolutely fast for the last 5 years. But anyway I would love to move to azure, because I don't have the time to manage updates and monitor everything. Even 100$ would be OK, as long as I know the server is stable and performs well. But I think that hosting at azure is only suitable for large companies which can save the need of one (or two) administrators. And in my case, azure is just too expensive.

I don't understand why Microsoft cannot/or don't want to have similar offers. This is the first question...
But what I understand even less, which people are using small azure instances and spend that money???

Colin Kness
October 01, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Your are so spot on. Performance is the worst on the VM doing nothing but starting a OS. They are charging you for all there over head to get the server functionality running. Where linux is just there boom fast.

The network vpn services just go down ..take forever to come back up.

You cant move a machine from one network group to another.

on and on ...

Vaclav Elias
October 13, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

Thanks Rick, I did also my homework, did a few tests today comparing UK Fasthosts Cloud servers with similar configuration on Azure and can confirm that Azure VM is soooo slower. I put my comment here "Reduce cloud costs.." on Azure User Voice http://feedback.azure.com/forums/34192--general-feedback/suggestions/1823899-reduce-cloud-costs-specifically-storage-and-bandw as to get the same performance I would need to spend twice more on Azure VM...

At the moment, performance wise simply I wouldn't just recommend Azure VM unless somebody wants to use other services & features which are all good.

David
November 26, 2015

# re: Azure VM Blues: Fighting a losing Performance Battle

I originally had a similar performance problem to those mentioned above.

I ended up moving my VMs to the Brasil South data centre, and was pleasantly surprised to find an immediate improvement in speed compared to other locations. Comparing a D2 instance freshly created in East US2, South Central US, and Brasil South, with the exact same Data drive consisting of four vhds combined via Storage Spaces, Brasil South is considerably faster. The downside is the internet access to MSDN etc is considerably slower, not all services are available, and not everyone has access to it.

It seems to be that as Brasil South is one of the newer data centres, it has later generation Xeons (they have 20 MB caches).

I also tried DS series machines in East US2, with Premium Storage, and as you would expect, they are *much* faster than the other non SSD based VMs - however the cost of Premium Storage ate through my monthly budget in two weeks.

Chris
January 27, 2016

# re: Azure VM Blues: Fighting a losing Performance Battle

Why use VMs at all? It doesn't sound like you need a full Server environment to host custom applications. If all you are hosting are web sites and web/db apps use the PaaS rather than IaaS parts of Azure. Put a little time into configuring you web sites into Azure Web Apps or Azure App Services. Cheaper, faster and nearly no administration.

Todd S.
February 13, 2016

# re: Azure VM Blues: Fighting a losing Performance Battle

Such a wealth of information, thank you for this post! Especially the tip on "Don’t use Quick Create".

Ali
February 28, 2016

# re: Azure VM Blues: Fighting a losing Performance Battle

Just wanted to throw in my 2c after having almost exactly the same experience. I've been using Vultr for a while - specifically I regularly deploying instances for test/dev purposes, or for batch jobs requiring fast internet (I'm often on 4G). When I finally had an application for Azure, I assumed I'd be getting similar/decent levels of performance. With my tiny workload, I started with an A0 instance, but found performance to be unbearable on anything less than a D3. Even then, the simple umbraco site was performing slowly, the VM seemed slower than Vultr, and my install script took me a day to go through.

So after going through the full Azure infrastructure set-up and prod deployment, I had to give up and switched them to Vultr. On a 1GB instance, I ran through half my install script in 15 mins. Filled up the drive, scaled out, to 2GB (45GB SSD) and finished installing everything in an around an hour. Everything is just absolutely blazing fast. The machines even reboots in under 15 seconds.

Walking away just stunned that the blokes who wrote the OS and the hypervisor are so damned far behind everyone else, and cursing at Azure and Microsoft for wasting billable hours through their over-provisioning. Cost-savings aside, Azure wasted my time, and I feel pretty bad billing a customer because I was foolish enough to believe Microsoft would actually deliver what they promised. After looking around the 'net I'm really shocked at just how terrible Azure really is. Even the PaaS stuff has serious perf issues. So it'll be a long while before I bother looking at Azure again.

Anyway, my next trick (in my spare time) will be to do this with an ISO-based install from MSDN. I'm curious to see how low I can go... :)

Sean Cleaver
March 26, 2016

# re: Azure VM Blues: Fighting a losing Performance Battle

Thought it was just me, thank you for the great detailed post. I tried Azure a few years back to host our website and an attempt at a SaaS application. I used WebSites and SQL Azure and a few VM's I found that the site was always going down despite the whole idea that it was always up, the SQL Azure DB's would just fail to connect every now and then and the VM's were just not up to the task. I decided that to use Azure you need to ensure you use multiple servers and zones for everything and replicate your data to get any kind of reliable uptime. This is Waaaaayyy too expensive for small sites. The changes to SQL Azure for DTU based pricing makes SQL really expensive when you start getting a few DB's especially when there tiny.

I moved over to Amazon EC2 and SQL RDS been running for about 18 months like this with no downtime it was far cheaper too with just a t2.small and t2.micro SQL Express Instance the total cost is about the same as 1 Azure VM.

Amazon RDS is great you get a SQL Express instance that you can install multiple small DB's on with automated backups that just runs.

The Desktop under Amazon EC2 is still a little sluggish (I couldn't DEV from it) but nothing like Azure at least you can use it.

Setting up a Private VLan in Amazon is easy, were as in Azure I could not find how to keep my SQL Azure DB's private. You have IP filtering but that's no good with Websites and you can block the internet but doesn't stop anybody with an Azure account.

I have tried Azure again this weekend since I really would like it to work but the VM just feel far too slow you can't do anything from the desktop. So I give up I will stick with Amazon and RDS it's not too expansive and works really well.
 

West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2018