I was doing some work testing one of my older ISAPI extensions under IIS 7. One of the tests I run purposefully terminates the process to simulate a hard crash in the ISAPI process. This should never happen but as we all know from time to time things go wrong to the point where the process can’t recover and fails (usually related to running out of memory in some way).
Anyway – IIS 6 and 7 both use separate worker processes to handle all request processing and there’s monitoring built into the IISADMIN service that detects the crash and automatically restarts the application after a hard crash...
Uh, that’s the way it’s supposed to work <s>. IIS 7 introduces a new feature called Rapid Failure Protection which is on by default. What it does, it checks for a specified number of hard failures like the ones I was testing in a given time period. The default is 5 failures in 5 minutes after which the Application Pool will crash and not restart automatically.
After a failure of this sort you end up with a stopped application pool which I talked about a couple of weeks back and which looks like this:
When this is the case the server returns a 503.2 – Service unavailable error.
When the Application Pool is shut down in this fashion an entry is written to the Event Log. BTW, IIS 7 is MUCH, MUCH better about reporting errors properly so if something goes wrong with IIS be sure to check the System event log. I've found many an error in there and unlike the old IIS error messages there that were pretty much worthless beyond "I bonked - good luck finding what happened", the new entries in the event log are quite useful and most likely will point you right at the problem. I have to train myself for that actually as my first reaction to the above problem was to post a message on the IIS forums <s>...
Anyway, I suppose this is a useful feature, just a little unexpected in its default configuration. The idea is that if you have a service that continues to crash you probably don’t want to have it going up and down forever and consuming resources and potentially corrupting data as it keeps failing. However, I think the timeout interval and failure mode is maybe a little too long. 5 failures in ONE minute would maybe be better as it would likely rule out debugging scenarios.
Ntoice that you can also specify an EXE to run when the Application Pool is shut down. This might allow you to send an email notification or some other sort of reminder to let you know right now that something's wrong. It might also allow you to take the machine out of a load balancing cluster if that's possible externally. Very cool. It's nice to see some of the new features have been thought through well to provide some very useful functionality.
IIS 7 features many new administration features like this and while useful it's going to take some time to get a handle on the subtle ways that things have changed in the default configuration.
Other Posts you might also like