After some extensive debugging and some help from the ASP.Net team I have found the problem for the Mysterious System.ExecutionEngineException failure with the ASP.Net Runtime  problem I logged about a while ago.

What is happening is that inside of the application multiple versions of the ASP.Net runtime are getting loaded, so that both the ASP. Net 1.1 and 2.0 versions of the ASPNET_ISAPI.DLL live in the current process. The failure occurs if the running application picks the wrong version of the two loaded into memory.

It turns out that these instances of the ASPNET_ISAPI.DLL get loaded through System.Diagnostics. Specifically my Windows Forms app calls a method to reduce the Working Set of the app whenever the main window is deactivated. This code looks like this and uses the Process class:

public static void SetWorkingSet(int lnMaxSize,int lnMinSize)

{

      System.Diagnostics.Process loProcess =       System.Diagnostics.Process.GetCurrentProcess();

      loProcess.MaxWorkingSet = (IntPtr) lnMaxSize;

      loProcess.MinWorkingSet = (IntPtr) lnMinSize;

      long lnValue = loProcess.WorkingSet;  // left in for debugging by accident

}

If you run the form under Native and CLR debugging you will find that the last line that reads the WorkingSet, causes all sorts of  crap to get loaded into the current process. Crap indeed: Take a look at the list of DLLs that get loaded:

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\netfxperf.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\PerfCounter.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\ntmarta.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\wldap32.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\samlib.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\CORPerfMonExt.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\psapi.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\aspperf.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\Microsoft.NET\Framework\v2.0.40607\aspnet_isapi.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\Microsoft.NET\Framework\v2.0.40607\msvcr80.dll', Cannot find or open the PDB file.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\userenv.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\Microsoft.NET\Framework\v1.1.4322\aspnet_isapi.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\query.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\crypt32.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\msasn1.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\infoctrs.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\infoadmn.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\iisRtl.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\ws2_32.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\ws2help.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\winipsec.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\msdtcuiu.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\atl.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\mfc42u.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\msvcp60.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\msdtcprx.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\netapi32.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\mtxclu.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\version.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\wsock32.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\comres.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\clusapi.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\resutils.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\mqperf.dll', No symbols loaded.

'WebStoreClient.exe': Unloaded 'D:\WINDOWS\system32\mqperf.dll'

'WebStoreClient.exe': Loaded 'D:\Program Files\Common Files\Microsoft Shared\MSSEARCH\BIN\msscntrs.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\programs\MSSQL\Binn\sqlctr80.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\snprfdll.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\staxmem.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\Program Files\Common Files\System\MSMAPI\1033\MSMAPI32.DLL', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\Program Files\Common Files\Microsoft Shared\OFFICE11\MSO.DLL', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\perfdisk.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\perfnet.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\perfos.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\perfproc.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\pop3server\pop3perf.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\loadperf.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\pschdprf.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\traffic.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\iphlpapi.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\wmi.dll', No symbols loaded.

'WebStoreClient.exe': Unloaded 'D:\WINDOWS\system32\pschdprf.dll'

'WebStoreClient.exe': Unloaded 'D:\WINDOWS\system32\traffic.dll'

'WebStoreClient.exe': Unloaded 'D:\WINDOWS\system32\wmi.dll'

'WebStoreClient.exe': Unloaded 'D:\WINDOWS\system32\iphlpapi.dll'

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\rasctrs.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\rasman.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\secur32.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\msv1_0.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\smtpctrs.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\smtpapi.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\winspool.drv', No symbols loaded.

First-chance exception at 0x77e649d3 in WebStoreClient.exe: 0x000006B5: The interface is unknown.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\tapiperf.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\tapi32.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\rtutils.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\winmm.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\Echo24Wrap.dll', Symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\Echo24Wave.dll', No symbols loaded.

'WebStoreClient.exe': Unloaded 'D:\WINDOWS\system32\Echo24Wave.dll'

'WebStoreClient.exe': Unloaded 'D:\WINDOWS\system32\Echo24Wrap.dll'

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\Echo24Wrap.dll', Symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\Echo24Wave.dll', No symbols loaded.

'WebStoreClient.exe': Unloaded 'D:\WINDOWS\system32\Echo24Wave.dll'

'WebStoreClient.exe': Unloaded 'D:\WINDOWS\system32\Echo24Wrap.dll'

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\perfctrs.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\iphlpapi.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\mprapi.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\activeds.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\adsldpc.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\credui.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\setupapi.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\perfts.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\winsta.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\utildll.dll', No symbols loaded.

First-chance exception at 0x77e649d3 in WebStoreClient.exe: 0x000006B5: The interface is unknown.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\inetsrv\w3ctrs.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\inetsrv\iisutil.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\wbem\wmiaprpl.dll', No symbols loaded.

'WebStoreClient.exe': Loaded 'D:\WINDOWS\system32\wbem\wbemcomn.dll', No symbols loaded.

Quite a list, huh? PO3Perf, IISPerf - heck even my Audio drivers get loaded.

The real culprit for the ASP.Net runtime hosting, turns out to be the PerfCounter Dlls which apparently load up some code in ASPNET_ISAPI.DLL. Problem is that the codepath here loads two versions of the DLL – both the 2.0 and 1.1 version. The new hosting process tries to use the 2.0 version if that one was loaded into the current process.

This was one hell of a bug to find and the key here was a hint from Gjergji Stasa of the ASP.Net team who looked at the DLL Load list and guessed that the perf counters were at fault. After that and some additional debugging using the Unmanaged debugging in VS.Net to trace all the DLLs that were getting loaded it was relatively easy to trace down where the perf stuff was getting loaded.

This also explains the flakey behavior I was getting with some runtime host requests failing and requests always working after the first one. The behavior I was seeing only occurred if the SetWorkingSet was called prior to the first time the Runtime Host was activated. After that it always worked.

The real moral of the story is: We need to be wary sometimes of what gets loaded for us behind the scenes by the runtime. It seems very, very odd that reading Process.WorkingSet should cause all of these extra DLLs to get loaded. In my code above I had left in that value merely as a debugging aid. To fix all I needed to do is remove the line that read the working set. What’s really strange is that SETTING the working set works just fine – the reading is what’s causing the problem. Go figure...