Rick Strahl's Weblog  

Wind, waves, code and everything in between...
.NET • C# • Markdown • WPF • All Things Web
Contact   •   Articles   •   Products   •   Support   •   Advertise
Sponsored by:
West Wind WebSurge - Rest Client and Http Load Testing for Windows

IIS 7 and Rapid Fail Protection


:P
On this page:

I was doing some work testing one of my older ISAPI extensions under IIS 7. One of the tests I run purposefully terminates the process to simulate a hard crash in the ISAPI process. This should never happen but as we all know from time to time things go wrong to the point where the process can’t recover and fails (usually related to running out of memory in some way).

 

Anyway – IIS 6 and 7 both use separate worker processes to handle all request processing and there’s monitoring built into the IISADMIN service that detects the crash and automatically restarts the application after a hard crash...

 

Uh, that’s the way it’s supposed to work <s>. IIS 7 introduces a new feature called Rapid Failure Protection which is on by default. What it does, it checks for a specified number of hard failures like the ones I was testing in a given time period. The default is 5 failures in 5 minutes after which the Application Pool will crash and not restart automatically.

  

 

After a failure of this sort you end up with a stopped application pool which I talked about a couple of weeks back and which looks like this:

 

 

 

When this is the case the server returns a 503.2 – Service unavailable error.

 

When the Application Pool is shut down in this fashion an entry is written to the Event Log. BTW, IIS 7 is MUCH, MUCH better about reporting errors properly so if something goes wrong with IIS be sure to check the System event log. I've found many an error in there and unlike the old IIS error messages there that were pretty much worthless beyond "I bonked - good luck finding what happened", the new entries in the event log are quite useful and most likely will point you right at the problem. I have to train myself for that actually as my first reaction to the above problem was to post a message on the IIS forums <s>...

  

Anyway, I suppose this is a useful feature, just a little unexpected in its default configuration. The idea is that if you have a service that continues to crash you probably don’t want to have it going up and down forever and consuming resources and potentially corrupting data as it keeps failing. However, I think the timeout interval and failure mode is maybe a little too long. 5 failures in ONE minute would maybe be better as it would likely rule out debugging scenarios.

 

Ntoice that you can also specify an EXE to run when the Application Pool is shut down. This might allow you to send an email notification or some other sort of reminder to let you know right now that something's wrong. It might also allow you to take the machine out of a load balancing cluster if that's possible externally. Very cool. It's nice to see some of the new features have been thought through well to provide some very useful functionality.

 

IIS 7 features many new administration features like this and while useful it's going to take some time to get a handle on the subtle ways that things have changed in the default configuration.

Posted in IIS  

The Voices of Reason


 

Martin Bennedik
December 20, 2006

# re: IIS 7 and Rapid Fail Protection

I think rapid fail protection was introduced with IIS 6, not 7.

Rick Strahl
December 20, 2006

# re: IIS 7 and Rapid Fail Protection

Ups, you're right Martin. The default is different apparently...

Interesting too - I think this may be the reason for my occasional problems with IIS not restarting properly some time back. I notice on IIS 6 all my Application Pools except the default one had this option off. So I suspect at some point Rapid Fail Protection kicked in and disabled the default App Pool and only a reboot allowed it to restart.

Rick Strahl's Web Log
June 24, 2007

# 503 Service Unavailable in IIS 7 - watch those Application Pools - Rick Strahl's Web Log

I ran into a funky issue with an application on IIS 7 today on my development machine: One of my Web applications simply would not respond to requests and IIS was reporting a 503 Error – Service Unavailable. Oddly enough though the rest of my Web site was working just fine – just this one...

Don
January 25, 2008

# re: IIS 7 and Rapid Fail Protection

What defines a "hard failure" which triggers Rapid Fail to start counting? I assume it's not things like, forgetting a semi-colon, or incorrectly casting a variable?

Doug
January 29, 2008

# re: IIS 7 and Rapid Fail Protection

Rick, i just came across this post, we are having a strange thing happen at work
our app pool is crashing, but it looks like it is being caused by an APP! running on the 2.0 app pool. No file permission errors in the log, just a shutdown message.

we are using IIS6, so this might not be the right place to put this..but it is causing major havoc on our production system (a lot of manual creating of orders, since our app isnt working..big headaches)

Do you know of a good way through c# to purposely crash the app pool, so we can at least have something to test with (I am almost certain it has to do with our threading..)??

thanks!

Cole
August 01, 2011

# re: IIS 7 and Rapid Fail Protection

I was debugging and couldn't figure out why I suddenly got 503. You saved the day! Thank you.

Cees meuleman
February 24, 2012

# re: IIS 7 and Rapid Fail Protection

The application pool of my production website was crashing once every so often.
The real issue I had was that I need to manually restart the pool and in the meantime my visitors would get an 503 error.
I used the shutdown executable option and parameters to have it automatically restart.

Shutdown executable: c:\Windows\System32\inetsrv\Appcmd.exe
Shutdown parameters: start APPPOOL <your website>

Alternatively you can choose to do an iisreset and if that fails a server reboot:
Shutdown executable: c:\Windows\System32\iisreset.exe
Shutdown parameters: /rebootonerror

Evan Reyes
December 06, 2019

# re: IIS 7 and Rapid Fail Protection

We had the same issue. However, the real problem we have is restarting the App Pool in IIS because upon Rapid Fail Protection triggered the disable. The App Pool does not seem that it stopped as it still showing as Started. When we try to restart it, it was notifying that it cannot be restarted and took an hour for it to restart upon keeping on trying to restart. Anyone know why this happens, the App pool not starting?


West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2024