ISAPI Module Crashing on Application Pool Shutdown

August 22, 2008 • from Maui, Hawaii • 15 comments

On this page:

So I have an odd issue with an ISAPI DLL that one of my products uses. The ISAPI interface provides a gateway interface to application servers and the actual module itself works fine without any problems. The problem is that the module is apparently crashing occasionally when the Application Pool is shutting down, but it's crashing after any of the DLL code has long stopped running.

However, on IIS7 (and in some cases on IIS6 as well), the ISAPI module is causing crashes that are showing up in the event log. The error in the event log ends up like this:

ISAPIBug

The errors are rare and only occur when the Application Pool is unloaded and even then not every time. It tends to require that the app's been running for a long time before I see the errors. The Application Pool is on auto-recyle every night and that why I end up with these entries in the Event Log, but there are other situations where the Application Pool gets restarted obviously.

The above is not really causing a problem - IIS just keeps on ticking, but a number of customers have seen the event log entries and complained mainly because it's not obvious whether these are operational failures or shut down failures.

The odd thing is that this happens on shut down after all user code is long done executing. I have cleanup code in TerminateExtension() as well as some last minute clean up code in the DLL's PROCESS_DETACH handler all of which runs without any problems when I enable logging and output. I also use COM Interop in this ISAPI extension and a number of COM servers are loaded and managed in a native thread pool. The extension can also run servers in a non-COM mode, in which case the failures go away. So it appears the problem is related to the COM server references possibly not getting cleaned up properly, but since the error is occurring completely outside of my application's domain (ie. after DLL_PROCESS_DETACH in DllMain) I have no way to see what's actually leading up to this crash.

I've attached a debugger and am trying to debug this issue, but the debugger won't hit on the failure either because it appears while the process in the middle of a shutdown.

I'm also pretty baffled by the error message. Notice that it talks about wc.dll_unloaded which appears to be some system generated placeholder file rather than my actual dll:

Faulting application w3wp.exe, version 7.0.6001.18000, time stamp 0x47919413, faulting module wc.dll_unloaded, version 0.0.0.0, time stamp 0x4441d309, exception code 0xc0000005, fault offset 0x0031e6ba, process id 0x7e0, application start time 0x01c9032a50903e70.

I did a bit of searching around for this issue and it looks like my extension isn't the only one that this is happening to - a number of other ISAPI connectors (some versions of PHP for example) also seem to be suffering similar problems.

Wondering if anybody has run into this issue before and maybe dealt with this before. I'm kind of stumped on this one - I don't even have a good idea how to further debug this other than go randomly hunting for COM reference cleanup. There are some tips on how to get a stack trace to try to isolate the component that's failing but by way of the above error message the component is pretty clear <s>. Question is why this weird DLL placeholder is bombing and why it's doing this AFTER the app is practically completely shut down. Even assuming some sort of hung COM reference - those types of errors typically occur prior to DLL_PROCESS_DETACH.

Stumped.

Incidentally I have a .NET module that basically uses a very similar approach to manage the COM Servers and it works without any problems on IIS shutdown. But the .NET code is also a heck of a lot cleaner than the horrendous ISAPI code. Unfortunately the .NET module is not a 100% solution because of the dreaded tie to an IIS virtual/Appdomain <shrug>

The Voices of Reason

Steven Berkovitz
August 22, 2008

# re: ISAPI Module Crashing on Application Pool Shutdown

ADPlus is your friend! Why not grab a dump using -crash and see whats up?

Rick Strahl
August 23, 2008

# re: ISAPI Module Crashing on Application Pool Shutdown

So I found the problem with this - it's a handle leak in COM Object creation and release. Apparently as soon as the handle leak occurs shutting down will result in the above error message.

It appears to be happening on any of the EXE servers instantiated used by the ISAPI extension. But I traced this down to the simple COM creation and .Release() call leaking a handle:

hr = CoCreateInstance(clsid, NULL, CLSCTX_SERVER, IID_IUnknown, (LPVOID *) &punkObj);        
punkObj->Release();        
punkObj = NULL;

When I run this code the COM server is properly loaded and unloaded (it can optionally be made visible and it does pop up on the desktop that way). When running the above code the server loads and unloads as can be seen in process manager.

So why the handle leak here?

Kevin Jankunas
September 06, 2008

# re: ISAPI Module Crashing on Application Pool Shutdown

I'm Assuming your calling CoInitialize(); and CoUninitialize();
I'm also Assumng the HRESULT returned from CoCreateInstance is not failing.

Looks more like something is happening in the proxy stub code or some other handle or resource is not getting cleaned up correctly. How is the ISAPI using the reference to punkObj?

Rick Strahl
September 06, 2008

# re: ISAPI Module Crashing on Application Pool Shutdown

Actually no I'm not calling CoInitialize() and CoUninitialize() - IIS does this when threads are started so I have to use the env that IIS provides. But I've duplicated the same situation locally in a console application.

http://www.west-wind.com/Weblog/posts/463120.aspx

It's fairly reliably leaking Windows handles with most EXE based com servers. Still waiting to hear back from a contact at MS who's duplicated the issue and is trying to find out more. Apparently there's not much interest at MS at looking at a COM issue like this.

Not surprised. <s>

Kevin Jankunas
September 06, 2008

# re: ISAPI Module Crashing on Application Pool Shutdown

Looks like something funky in vfp. I downloaded the sample and I'm getting the same results. I thought adding an explicit call to Quit on the IApplication interface before calling Release() might work, but the proxy still leaks the handle.

Rick Strahl
September 07, 2008

# re: ISAPI Module Crashing on Application Pool Shutdown

@Kevin - Thanks for downloading and checking this out. VFP definitely leaks, but I've tried a bunch of other servers too and they too leak in the same way. The only things that appear not to leak are really light weight ATL components. BUt heavier servers - especially those that have some sort of UI all leak.

Reberto
October 31, 2008

# re: ISAPI Module Crashing on Application Pool Shutdown

Hi

I have same problem and i have been searching 4 days to solve it, but I have not

I had a c++ dll that worked when i added it to isapi-filters under win 2003 or xp under iis. but when i add it to iis7 (vista) I get same error as you.

Have you found a solution? let me know please

thanks alot.

Jacques Herweijer
May 18, 2009

# re: ISAPI Module Crashing on Application Pool Shutdown

Hi Rick, any news on this subject yet, seem to have stumbled this error too on a 64bit OS, maybe that has something to do with it, cause wc.dll is 32bit ? just a thought.

I'm using your webconnection wc.dll and at the production site I see the behavior happening at runtime. The request gets handled and then it is hanging (showstatus page shows me active seconds increasing), waiting to timeout at the number of seconds set in wc.ini. the processID becomes 0, restarts itself and event message is written to eventlog aswell as the ISAPI errorlog (txtfile).

Is this fixed in an update of your Webconnection product yet or is there some configuration that has to be adjusted ?

Rick Strahl
May 18, 2009

# re: ISAPI Module Crashing on Application Pool Shutdown

@Jacques - the problem I'm describing above is non-destructive - other than the event log entry it has really no negative effect on the application or anything else because the Web Connection DLL and COM servers are long shut down by the time the error occurs deep down in the bowels of Windows.

What you're describing is something different and likely has nothing to do with this particular problem.

If you have issues with 64bit make sure you follow the instructions in the help file for configuring your OS for/IIS for 32 bit operation. On IIS 6 this is not a trivial process, but on IIS 7 you can isolate the application pool. Recent versions of Web Connection have a few tweaks for operation on 64 bit servers, but nothing that addresses what you're experiencing. You seem to be suffering from hung servers (bad requests hanging possibly?).

Ghalib Ghniem
June 07, 2010

# re: ISAPI Module Crashing on Application Pool Shutdown

Hi,
Thank you for such information.
We face a similar problem..Our application is ASP and running in windows 2008 32 bit, IIS 7 with SQL 2008.
Each 10-20 minutes the pool crashes and restarts, there is 2 errors in the event viewer:

1. Faulting application w3wp.exe, version 7.0.6002.18005, time stamp 0x49e023cf, faulting module ntdll.dll, version 6.0.6002.18005, time stamp 0x49e03821, exception code 0xc00000fd, fault offset 0x0004a4d2, process id 0x1624, application start time 0x01cb068ad53c5f95.

2. The application-specific permission settings do not grant Local Activation permission for the COM Server application with CLSID
{0C0A3666-30C9-11D0-8F20-00805F2CD064}
to the user IIS APPPOOL\menafn SID (S-1-5-82-3702171470-1990676222-2752937436-66822189-843242211) from address LocalHost (Using LRPC). This security permission can be modified using the Component Services administrative tool.

Is there any idea

_BrianCruz
October 26, 2010

# re: ISAPI Module Crashing on Application Pool Shutdown

Hi,

I have the same configuration, and errors, like Ghalib. I'm trying to get the FirstData PaymentGateway up and running with AspDotNetStorefront but when going through the cart it hangs and the appPool restarts. I also applied other updated DLL files that were giving to me from FirstData, but I believe it's something more than that. Has anyone happen to figure out how to correct this problem, especially you Ghalib. Thanks in advance...

_BrianCruz

Kev
March 07, 2011

# re: ISAPI Module Crashing on Application Pool Shutdown

I also have the same error as Ghalib.

Faulting application w3wp.exe, version 7.0.6002.18005, time stamp 0x49e023cf, faulting module ntdll.dll, version 6.0.6002.18327, time stamp 0x4cb73436, exception code 0xc0000005, fault offset 0x000665d6, process id 0x974, application start time 0x01cbdcedcd953cee.

We also run ASP and some .NET (VB) on Windows Server 2008, 32 bit, IIS 7, with SQL 2008.

This crash does not hang anything.

This even appears alone, or sometimes twice in a row, but does not seem to be associated directly with other errors we receive.

Richard Algeni Jr
June 02, 2019

# re: ISAPI Module Crashing on Application Pool Shutdown

You might not believe this, but I am having the same problem Rick, using Windows Server 2019 and IIS 10. After my ISAPI extension is unloaded, either by timeout (21 minutes of inactivity) or by the command iisreset, I get the Exception Code: c0000005.

The last thing the extension does in the TerminateExtension function is cleanup some pooled connections to a Redis server, create a log entry, and create an event entry. All of those work fine. However, I commented all of this out, and the error remains. I am writing classic dll's using PureBasic, 64 bit unicode. It does not seem to cause any issues, other than annoying the Hell out of me as I prefer to see clean event logs.

Source:        iisapi_caddata.dll
Date:          6/2/2019 4:33:03 PM
Event ID:      801
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      Andromeda.OrionTEK.local
Description:
TerminateExtension() > Process terminating, all resources have been released
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="iisapi_caddata.dll" />
    <EventID Qualifiers="0">801</EventID>
    <Level>4</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2019-06-02T20:33:03.520641100Z" />
    <EventRecordID>21718</EventRecordID>
    <Channel>Application</Channel>
    <Computer>Andromeda.OrionTEK.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>TerminateExtension() &gt; Process terminating, all resources have been released</Data>
  </EventData>
</Event>```

```Log Name:      Application
Source:        Application Error
Date:          6/2/2019 4:33:03 PM
Event ID:      1000
Task Category: (100)
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      Andromeda.OrionTEK.local
Description:
Faulting application name: w3wp.exe, version: 10.0.17763.1, time stamp: 0xcfdb13d8
Faulting module name: iisapi_caddata.dll_unloaded, version: 1.0.0.0, time stamp: 0x5cf310bf
Exception code: 0xc0000005
Fault offset: 0x0000000000034230
Faulting process id: 0x3240
Faulting application start time: 0x01d5197f70f26837
Faulting application path: c:\windows\system32\inetsrv\w3wp.exe
Faulting module path: iisapi_caddata.dll
Report Id: 56c79e65-df1a-498a-8e28-339ab06f3f3b
Faulting package full name: 
Faulting package-relative application ID: 
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="Application Error" />
    <EventID Qualifiers="0">1000</EventID>
    <Level>2</Level>
    <Task>100</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2019-06-02T20:33:03.563640000Z" />
    <EventRecordID>21719</EventRecordID>
    <Channel>Application</Channel>
    <Computer>Andromeda.OrionTEK.local</Computer>
    <Security />
  </System>
  <EventData>
    <Data>w3wp.exe</Data>
    <Data>10.0.17763.1</Data>
    <Data>cfdb13d8</Data>
    <Data>iisapi_caddata.dll_unloaded</Data>
    <Data>1.0.0.0</Data>
    <Data>5cf310bf</Data>
    <Data>c0000005</Data>
    <Data>0000000000034230</Data>
    <Data>3240</Data>
    <Data>01d5197f70f26837</Data>
    <Data>c:\windows\system32\inetsrv\w3wp.exe</Data>
    <Data>iisapi_caddata.dll</Data>
    <Data>56c79e65-df1a-498a-8e28-339ab06f3f3b</Data>
    <Data>
    </Data>
    <Data>
    </Data>
  </EventData>
</Event>```

Richard Algeni Jr
June 09, 2019

# re: ISAPI Module Crashing on Application Pool Shutdown

Update: After downloading and installing DebugDiag 1.2 from Microsoft, and analyzing the crash dump, the access violation has stopped. Everything else I had done up to that point had made difference.

Johnny
February 28, 2024

# re: ISAPI Module Crashing on Application Pool Shutdown

@Richard Algeni Jr And what was it you did up to this point to solve it?

Rick Strahl's Weblog