Contact   •   Products   •   Search

Rick Strahl's Web Log

Wind, waves, code and everything in between...
ASP.NET • C# • HTML5 • JavaScript • AngularJs

ISAPI Module Crashing on Application Pool Shutdown


So I have an odd issue with an ISAPI DLL that one of my products uses. The ISAPI interface provides a gateway interface to application servers and the actual module itself works fine without any problems. The problem is that the module is apparently crashing occasionally when the Application Pool is shutting down, but it's crashing after any of the DLL code has long stopped running.

However, on IIS7 (and in some cases on IIS6 as well), the ISAPI module is causing crashes that are showing up in the event log. The error in the event log ends up like this:

ISAPIBug

The errors are rare and only occur when the Application Pool is unloaded and even then not every time. It tends to require that the app's been running for a long time before I see the errors. The Application Pool is on auto-recyle every night and that why I end up with these entries in the Event Log, but there are other situations where the Application Pool gets restarted obviously.

The above is not really causing a problem - IIS just keeps on ticking, but a number of customers have seen the event log entries and complained mainly because it's not obvious whether these are operational failures or shut down failures.

The odd thing is that this happens on shut down after all user code is long done executing. I have cleanup code in TerminateExtension() as well as some last minute clean up code in the DLL's PROCESS_DETACH handler all of which runs without any problems when I enable logging and output. I also use COM Interop in this ISAPI extension and a number of COM servers are loaded and managed in a native thread pool. The extension can also run servers in a non-COM mode, in which case the failures go away. So it appears the problem is related to the COM server references possibly not getting cleaned up properly, but since the error is occurring completely outside of my application's domain (ie. after DLL_PROCESS_DETACH in DllMain) I have no way to see what's actually leading up to this crash.

I've attached a debugger and am trying to debug this issue, but the debugger won't hit on the failure either because it appears while the process in the middle of a shutdown.

I'm also pretty baffled by the error message. Notice that it talks about wc.dll_unloaded which appears to be some system generated placeholder file rather than my actual dll:

Faulting application w3wp.exe, version 7.0.6001.18000, time stamp 0x47919413, faulting module wc.dll_unloaded, version 0.0.0.0, time stamp 0x4441d309, exception code 0xc0000005, fault offset 0x0031e6ba, process id 0x7e0, application start time 0x01c9032a50903e70.

I did a bit of searching around for this issue and it looks like my extension isn't the only one that this is happening to - a number of other ISAPI connectors (some versions of PHP for example) also seem to be suffering similar problems.

Wondering if anybody has run into this issue before and maybe dealt with this before. I'm kind of stumped on this one - I don't even have a good idea how to further debug this other than go randomly hunting for COM reference cleanup. There are some tips on how to get a stack trace to try to isolate the component that's failing but by way of the above error message the component is pretty clear <s>. Question is why this weird DLL placeholder is bombing and why it's doing this AFTER the app is practically completely shut down. Even assuming some sort of hung COM reference - those types of errors typically occur prior to DLL_PROCESS_DETACH.

Stumped.

Incidentally I have a .NET module that basically uses a very similar approach to manage the COM Servers and it works without any problems on IIS shutdown. But the .NET code is also a heck of a lot cleaner than the horrendous ISAPI code. Unfortunately the .NET module is not a 100% solution because of the dreaded tie to an IIS virtual/Appdomain <shrug>

Make Donation
Posted in C++  IIS  


Feedback for this Post

 
# re: ISAPI Module Crashing on Application Pool Shutdown
by Steven Berkovitz August 22, 2008 @ 6:04pm
ADPlus is your friend! Why not grab a dump using -crash and see whats up?
# re: ISAPI Module Crashing on Application Pool Shutdown
by Rick Strahl August 23, 2008 @ 12:54am
So I found the problem with this - it's a handle leak in COM Object creation and release. Apparently as soon as the handle leak occurs shutting down will result in the above error message.

It appears to be happening on any of the EXE servers instantiated used by the ISAPI extension. But I traced this down to the simple COM creation and .Release() call leaking a handle:

hr = CoCreateInstance(clsid, NULL, CLSCTX_SERVER, IID_IUnknown, (LPVOID *) &punkObj);        
punkObj->Release();        
punkObj = NULL;


When I run this code the COM server is properly loaded and unloaded (it can optionally be made visible and it does pop up on the desktop that way). When running the above code the server loads and unloads as can be seen in process manager.

So why the handle leak here?
# re: ISAPI Module Crashing on Application Pool Shutdown
by Xun Ding August 23, 2008 @ 6:11am
You write faster than I read. Incredible. Experience to thoughts, thoughts to words, spice it up with images, everything takes time, does it not?

Great blog!
# re: ISAPI Module Crashing on Application Pool Shutdown
by Kevin Jankunas September 06, 2008 @ 9:40pm
I'm Assuming your calling CoInitialize(); and CoUninitialize();
I'm also Assumng the HRESULT returned from CoCreateInstance is not failing.

Looks more like something is happening in the proxy stub code or some other handle or resource is not getting cleaned up correctly. How is the ISAPI using the reference to punkObj?
# re: ISAPI Module Crashing on Application Pool Shutdown
by Rick Strahl September 06, 2008 @ 10:04pm
Actually no I'm not calling CoInitialize() and CoUninitialize() - IIS does this when threads are started so I have to use the env that IIS provides. But I've duplicated the same situation locally in a console application.

http://www.west-wind.com/Weblog/posts/463120.aspx

It's fairly reliably leaking Windows handles with most EXE based com servers. Still waiting to hear back from a contact at MS who's duplicated the issue and is trying to find out more. Apparently there's not much interest at MS at looking at a COM issue like this.

Not surprised. <s>
# re: ISAPI Module Crashing on Application Pool Shutdown
by Kevin Jankunas September 06, 2008 @ 11:40pm
Looks like something funky in vfp. I downloaded the sample and I'm getting the same results. I thought adding an explicit call to Quit on the IApplication interface before calling Release() might work, but the proxy still leaks the handle.
# re: ISAPI Module Crashing on Application Pool Shutdown
by Rick Strahl September 07, 2008 @ 12:02am
@Kevin - Thanks for downloading and checking this out. VFP definitely leaks, but I've tried a bunch of other servers too and they too leak in the same way. The only things that appear not to leak are really light weight ATL components. BUt heavier servers - especially those that have some sort of UI all leak.
# re: ISAPI Module Crashing on Application Pool Shutdown
by Reberto October 31, 2008 @ 4:35am
Hi

I have same problem and i have been searching 4 days to solve it, but I have not

I had a c++ dll that worked when i added it to isapi-filters under win 2003 or xp under iis. but when i add it to iis7 (vista) I get same error as you.

Have you found a solution? let me know please

thanks alot.
# re: ISAPI Module Crashing on Application Pool Shutdown
by Jacques Herweijer May 18, 2009 @ 4:05pm
Hi Rick, any news on this subject yet, seem to have stumbled this error too on a 64bit OS, maybe that has something to do with it, cause wc.dll is 32bit ? just a thought.

I'm using your webconnection wc.dll and at the production site I see the behavior happening at runtime. The request gets handled and then it is hanging (showstatus page shows me active seconds increasing), waiting to timeout at the number of seconds set in wc.ini. the processID becomes 0, restarts itself and event message is written to eventlog aswell as the ISAPI errorlog (txtfile).

Is this fixed in an update of your Webconnection product yet or is there some configuration that has to be adjusted ?
# re: ISAPI Module Crashing on Application Pool Shutdown
by Rick Strahl May 18, 2009 @ 4:49pm
@Jacques - the problem I'm describing above is non-destructive - other than the event log entry it has really no negative effect on the application or anything else because the Web Connection DLL and COM servers are long shut down by the time the error occurs deep down in the bowels of Windows.

What you're describing is something different and likely has nothing to do with this particular problem.

If you have issues with 64bit make sure you follow the instructions in the help file for configuring your OS for/IIS for 32 bit operation. On IIS 6 this is not a trivial process, but on IIS 7 you can isolate the application pool. Recent versions of Web Connection have a few tweaks for operation on 64 bit servers, but nothing that addresses what you're experiencing. You seem to be suffering from hung servers (bad requests hanging possibly?).
# re: ISAPI Module Crashing on Application Pool Shutdown
by Ghalib Ghniem June 07, 2010 @ 3:02pm
Hi,
Thank you for such information.
We face a similar problem..Our application is ASP and running in windows 2008 32 bit, IIS 7 with SQL 2008.
Each 10-20 minutes the pool crashes and restarts, there is 2 errors in the event viewer:

1. Faulting application w3wp.exe, version 7.0.6002.18005, time stamp 0x49e023cf, faulting module ntdll.dll, version 6.0.6002.18005, time stamp 0x49e03821, exception code 0xc00000fd, fault offset 0x0004a4d2, process id 0x1624, application start time 0x01cb068ad53c5f95.

2. The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID
{0C0A3666-30C9-11D0-8F20-00805F2CD064}
to the user IIS APPPOOL\menafn SID (S-1-5-82-3702171470-1990676222-2752937436-66822189-843242211) from address LocalHost (Using LRPC). This security permission can be modified using the Component Services administrative tool.

Is there any idea
# re: ISAPI Module Crashing on Application Pool Shutdown
by _BrianCruz October 26, 2010 @ 11:22am
Hi,

I have the same configuration, and errors, like Ghalib. I'm trying to get the FirstData PaymentGateway up and running with AspDotNetStorefront but when going through the cart it hangs and the appPool restarts. I also applied other updated DLL files that were giving to me from FirstData, but I believe it's something more than that. Has anyone happen to figure out how to correct this problem, especially you Ghalib. Thanks in advance...

_BrianCruz
# re: ISAPI Module Crashing on Application Pool Shutdown
by Kev March 07, 2011 @ 1:02pm
I also have the same error as Ghalib.

Faulting application w3wp.exe, version 7.0.6002.18005, time stamp 0x49e023cf, faulting module ntdll.dll, version 6.0.6002.18327, time stamp 0x4cb73436, exception code 0xc0000005, fault offset 0x000665d6, process id 0x974, application start time 0x01cbdcedcd953cee.

We also run ASP and some .NET (VB) on Windows Server 2008, 32 bit, IIS 7, with SQL 2008.

This crash does not hang anything.

This even appears alone, or sometimes twice in a row, but does not seem to be associated directly with other errors we receive.
 


West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2014