So I have an odd issue with an ISAPI DLL that one of my products uses. The ISAPI interface provides a gateway interface to application servers and the actual module itself works fine without any problems. The problem is that the module is apparently crashing occasionally when the Application Pool is shutting down, but it's crashing after any of the DLL code has long stopped running.
However, on IIS7 (and in some cases on IIS6 as well), the ISAPI module is causing crashes that are showing up in the event log. The error in the event log ends up like this:
The errors are rare and only occur when the Application Pool is unloaded and even then not every time. It tends to require that the app's been running for a long time before I see the errors. The Application Pool is on auto-recyle every night and that why I end up with these entries in the Event Log, but there are other situations where the Application Pool gets restarted obviously.
The above is not really causing a problem - IIS just keeps on ticking, but a number of customers have seen the event log entries and complained mainly because it's not obvious whether these are operational failures or shut down failures.
The odd thing is that this happens on shut down after all user code is long done executing. I have cleanup code in TerminateExtension() as well as some last minute clean up code in the DLL's PROCESS_DETACH handler all of which runs without any problems when I enable logging and output. I also use COM Interop in this ISAPI extension and a number of COM servers are loaded and managed in a native thread pool. The extension can also run servers in a non-COM mode, in which case the failures go away. So it appears the problem is related to the COM server references possibly not getting cleaned up properly, but since the error is occurring completely outside of my application's domain (ie. after DLL_PROCESS_DETACH in DllMain) I have no way to see what's actually leading up to this crash.
I've attached a debugger and am trying to debug this issue, but the debugger won't hit on the failure either because it appears while the process in the middle of a shutdown.
I'm also pretty baffled by the error message. Notice that it talks about wc.dll_unloaded which appears to be some system generated placeholder file rather than my actual dll:
Faulting application w3wp.exe, version 7.0.6001.18000, time stamp 0x47919413, faulting module wc.dll_unloaded, version 0.0.0.0, time stamp 0x4441d309, exception code 0xc0000005, fault offset 0x0031e6ba, process id 0x7e0, application start time 0x01c9032a50903e70.
I did a bit of searching around for this issue and it looks like my extension isn't the only one that this is happening to - a number of other ISAPI connectors (some versions of PHP for example) also seem to be suffering similar problems.
Wondering if anybody has run into this issue before and maybe dealt with this before. I'm kind of stumped on this one - I don't even have a good idea how to further debug this other than go randomly hunting for COM reference cleanup. There are some tips on how to get a stack trace to try to isolate the component that's failing but by way of the above error message the component is pretty clear <s>. Question is why this weird DLL placeholder is bombing and why it's doing this AFTER the app is practically completely shut down. Even assuming some sort of hung COM reference - those types of errors typically occur prior to DLL_PROCESS_DETACH.
Incidentally I have a .NET module that basically uses a very similar approach to manage the COM Servers and it works without any problems on IIS shutdown. But the .NET code is also a heck of a lot cleaner than the horrendous ISAPI code. Unfortunately the .NET module is not a 100% solution because of the dreaded tie to an IIS virtual/Appdomain <shrug>