ISAPI Module Crashing on Application Pool Shutdown
So I have an odd issue with an ISAPI DLL that one of my products uses. The ISAPI interface provides a gateway interface to application servers and the actual module itself works fine without any problems. The problem is that the module is apparently crashing occasionally when the Application Pool is shutting down, but it's crashing after any of the DLL code has long stopped running.
However, on IIS7 (and in some cases on IIS6 as well), the ISAPI module is causing crashes that are showing up in the event log. The error in the event log ends up like this:
The errors are rare and only occur when the Application Pool is unloaded and even then not every time. It tends to require that the app's been running for a long time before I see the errors. The Application Pool is on auto-recyle every night and that why I end up with these entries in the Event Log, but there are other situations where the Application Pool gets restarted obviously.
The above is not really causing a problem - IIS just keeps on ticking, but a number of customers have seen the event log entries and complained mainly because it's not obvious whether these are operational failures or shut down failures.
The odd thing is that this happens on shut down after all user code is long done executing. I have cleanup code in TerminateExtension() as well as some last minute clean up code in the DLL's PROCESS_DETACH handler all of which runs without any problems when I enable logging and output. I also use COM Interop in this ISAPI extension and a number of COM servers are loaded and managed in a native thread pool. The extension can also run servers in a non-COM mode, in which case the failures go away. So it appears the problem is related to the COM server references possibly not getting cleaned up properly, but since the error is occurring completely outside of my application's domain (ie. after DLL_PROCESS_DETACH in DllMain) I have no way to see what's actually leading up to this crash.
I've attached a debugger and am trying to debug this issue, but the debugger won't hit on the failure either because it appears while the process in the middle of a shutdown.
I'm also pretty baffled by the error message. Notice that it talks about wc.dll_unloaded which appears to be some system generated placeholder file rather than my actual dll:
Faulting application w3wp.exe, version 7.0.6001.18000, time stamp 0x47919413, faulting module wc.dll_unloaded, version 0.0.0.0, time stamp 0x4441d309, exception code 0xc0000005, fault offset 0x0031e6ba, process id 0x7e0, application start time 0x01c9032a50903e70.
I did a bit of searching around for this issue and it looks like my extension isn't the only one that this is happening to - a number of other ISAPI connectors (some versions of PHP for example) also seem to be suffering similar problems.
Wondering if anybody has run into this issue before and maybe dealt with this before. I'm kind of stumped on this one - I don't even have a good idea how to further debug this other than go randomly hunting for COM reference cleanup. There are some tips on how to get a stack trace to try to isolate the component that's failing but by way of the above error message the component is pretty clear <s>. Question is why this weird DLL placeholder is bombing and why it's doing this AFTER the app is practically completely shut down. Even assuming some sort of hung COM reference - those types of errors typically occur prior to DLL_PROCESS_DETACH.
Stumped.
Incidentally I have a .NET module that basically uses a very similar approach to manage the COM Servers and it works without any problems on IIS shutdown. But the .NET code is also a heck of a lot cleaner than the horrendous ISAPI code. Unfortunately the .NET module is not a 100% solution because of the dreaded tie to an IIS virtual/Appdomain <shrug>
The Voices of Reason
# re: ISAPI Module Crashing on Application Pool Shutdown
Update: After downloading and installing DebugDiag 1.2 from Microsoft, and analyzing the crash dump, the access violation has stopped. Everything else I had done up to that point had made difference.
# re: ISAPI Module Crashing on Application Pool Shutdown
@Richard Algeni Jr And what was it you did up to this point to solve it?
# re: ISAPI Module Crashing on Application Pool Shutdown
# re: ISAPI Module Crashing on Application Pool Shutdown
It appears to be happening on any of the EXE servers instantiated used by the ISAPI extension. But I traced this down to the simple COM creation and .Release() call leaking a handle:
hr = CoCreateInstance(clsid, NULL, CLSCTX_SERVER, IID_IUnknown, (LPVOID *) &punkObj); punkObj->Release(); punkObj = NULL;
When I run this code the COM server is properly loaded and unloaded (it can optionally be made visible and it does pop up on the desktop that way). When running the above code the server loads and unloads as can be seen in process manager.
So why the handle leak here?
# re: ISAPI Module Crashing on Application Pool Shutdown
I'm also Assumng the HRESULT returned from CoCreateInstance is not failing.
Looks more like something is happening in the proxy stub code or some other handle or resource is not getting cleaned up correctly. How is the ISAPI using the reference to punkObj?
# re: ISAPI Module Crashing on Application Pool Shutdown
http://www.west-wind.com/Weblog/posts/463120.aspx
It's fairly reliably leaking Windows handles with most EXE based com servers. Still waiting to hear back from a contact at MS who's duplicated the issue and is trying to find out more. Apparently there's not much interest at MS at looking at a COM issue like this.
Not surprised. <s>
# re: ISAPI Module Crashing on Application Pool Shutdown
# re: ISAPI Module Crashing on Application Pool Shutdown
# re: ISAPI Module Crashing on Application Pool Shutdown
I have same problem and i have been searching 4 days to solve it, but I have not
I had a c++ dll that worked when i added it to isapi-filters under win 2003 or xp under iis. but when i add it to iis7 (vista) I get same error as you.
Have you found a solution? let me know please
thanks alot.
# re: ISAPI Module Crashing on Application Pool Shutdown
Thank you for such information.
We face a similar problem..Our application is ASP and running in windows 2008 32 bit, IIS 7 with SQL 2008.
Each 10-20 minutes the pool crashes and restarts, there is 2 errors in the event viewer:
1. Faulting application w3wp.exe, version 7.0.6002.18005, time stamp 0x49e023cf, faulting module ntdll.dll, version 6.0.6002.18005, time stamp 0x49e03821, exception code 0xc00000fd, fault offset 0x0004a4d2, process id 0x1624, application start time 0x01cb068ad53c5f95.
2. The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID
{0C0A3666-30C9-11D0-8F20-00805F2CD064}
to the user IIS APPPOOL\menafn SID (S-1-5-82-3702171470-1990676222-2752937436-66822189-843242211) from address LocalHost (Using LRPC). This security permission can be modified using the Component Services administrative tool.
Is there any idea
# re: ISAPI Module Crashing on Application Pool Shutdown
I'm using your webconnection wc.dll and at the production site I see the behavior happening at runtime. The request gets handled and then it is hanging (showstatus page shows me active seconds increasing), waiting to timeout at the number of seconds set in wc.ini. the processID becomes 0, restarts itself and event message is written to eventlog aswell as the ISAPI errorlog (txtfile).
Is this fixed in an update of your Webconnection product yet or is there some configuration that has to be adjusted ?
# re: ISAPI Module Crashing on Application Pool Shutdown
What you're describing is something different and likely has nothing to do with this particular problem.
If you have issues with 64bit make sure you follow the instructions in the help file for configuring your OS for/IIS for 32 bit operation. On IIS 6 this is not a trivial process, but on IIS 7 you can isolate the application pool. Recent versions of Web Connection have a few tweaks for operation on 64 bit servers, but nothing that addresses what you're experiencing. You seem to be suffering from hung servers (bad requests hanging possibly?).
# re: ISAPI Module Crashing on Application Pool Shutdown
I have the same configuration, and errors, like Ghalib. I'm trying to get the FirstData PaymentGateway up and running with AspDotNetStorefront but when going through the cart it hangs and the appPool restarts. I also applied other updated DLL files that were giving to me from FirstData, but I believe it's something more than that. Has anyone happen to figure out how to correct this problem, especially you Ghalib. Thanks in advance...
_BrianCruz
# re: ISAPI Module Crashing on Application Pool Shutdown
Faulting application w3wp.exe, version 7.0.6002.18005, time stamp 0x49e023cf, faulting module ntdll.dll, version 6.0.6002.18327, time stamp 0x4cb73436, exception code 0xc0000005, fault offset 0x000665d6, process id 0x974, application start time 0x01cbdcedcd953cee.
We also run ASP and some .NET (VB) on Windows Server 2008, 32 bit, IIS 7, with SQL 2008.
This crash does not hang anything.
This even appears alone, or sometimes twice in a row, but does not seem to be associated directly with other errors we receive.
# re: ISAPI Module Crashing on Application Pool Shutdown
You might not believe this, but I am having the same problem Rick, using Windows Server 2019 and IIS 10. After my ISAPI extension is unloaded, either by timeout (21 minutes of inactivity) or by the command iisreset, I get the Exception Code: c0000005.
The last thing the extension does in the TerminateExtension function is cleanup some pooled connections to a Redis server, create a log entry, and create an event entry. All of those work fine. However, I commented all of this out, and the error remains. I am writing classic dll's using PureBasic, 64 bit unicode. It does not seem to cause any issues, other than annoying the Hell out of me as I prefer to see clean event logs.