I've been working on replacing an ISAPI module which is a connector piece to an Application Server interface that interfaces with Visual FoxPro. In this interface the backend is managing a pool of COM objects that process requests and return results.

Dealing with COM objects might seem pretty passee in the age of .NET but nevertheless there's still a lot of that going around and I'm quite involved with this due to my FoxPro background and FoxPro's one interface mechanism to .NET being through COM. Anyhow, I essentially rebuild this ISAPI component for West Wind Web Connection in .NET in rather short order with great success.

The  connector is basically an interface that feeds request data from the Web server to an application server backend (a FoxPro application) and it can pass data between the connector and the server app in several different ways. One of these ways is via COM and the interface can work either with InProc our OutOfProc servers.

Now I know a few people are probably rolling their eyes at the thought of out of proc servers. However due to the nature of the interface and the ability to fully manage and shut down out of proc servers, out of proc servers are actually the preferred mechanism.

One big worry about this new handler interface has been performance compared to the ISAPI interface. This was one of things where there's no easy way to know beforehand what performance would look like so it took actually getting the COM pool manager into place before there was any way to see what performance looks like.

I was surprised to see that performance was actually slightly better than with the ISAPI extension. There are a number of reasons for that - one of which is the redesign of the pool manager using a set of threads that constantly stay active with the objects loaded on them. With the live objects on them (unlike COM Interop in ASP or ASP.NET) there's no reload overhead and unlike my original ISAPI extension there's no thread marshalling taking place. The server calls simply are routed to an available server on a separate thread.

So far so good.

STA for InProc and Out of Proc Servers

West Wind Web Connection primarily uses Out of Proc servers for various reasons. The idea is that servers are cached and kept alive so the performance difference between a DLL server and an EXE is not that drastic especially given the fact that 99% of processing time is spent in the actual FoxPro server code rather than in the connector code.

So when I originally built the component I built and tested with EXE servers - and got very respectable performance numbers from the HttpHandler. I'm running IIS 7 on Vista here and it doesn't seem like there are any limits (contrary to what Microsoft says) on the server as I was able to run straight ASPX pages through stress testing with well over 2000 requests a second (without any caching).

Performance of the HttpHandler turned out to be somewhat better than the ISAPI extension. I ran two sets of tests: One for raw COM interface performance by creating essentially a simple Helloworld method at the COM entry point, and another that runs through the full West Wind Web Connection framework and returns a full HTML page. The following are requests per second averaged out over 3  1 minute runs (right - it's just for rough over the thumb performance estimates <s>).

EXE Server

There's a big difference between the Raw COM call and the West Wind Web Connection basic response, because there's quite a bit of processing that occurs inside of the FoxPro server to set up for the request. Any amount of Fox code will drag these numbers down from the low level ISAPI/HttpHandler code. However, getting over 100 requests a second is still quite impressive.

What is nice is that the HttpHandler is actually a bit faster, although I noticed that the ISAPI version is less CPU intensive - apparently ASP.NET incurs a bit more overhead just getting to the actual handler code. The EXE server doesn't care about STA or MTA - it works the same either way since it runs out of process  and so doesn't need to used shared apartment thread state. This is pretty damn good performance for calling an EXE server on a heavy object like a Visual FoxPro COM object.

Now I can take the same EXE server and recompile it as a DLL server and run these same tests again against the HttpHandler, and here's where things get very weird:

DLL Server w/HttpHandler

The tests are basically set up with the COM Pool running either MTA threads or STA threads, and then running MTA threads with COM+ as either an Inproc Library or Out of Proc Application.

I ran  MTA mode for the DLL merely for raw perf comparison. MTA threads are not an option for an STA component (like VFP) as it can cause major corruption of the COM server. These tests didn't stress Visual FoxPro much and so there was no corruption but data access in particular will eventually show issues with things like misplaced record pointers and any public variable overlap. Not an option but I ran it here to see the difference in performance that one could expect with an MTA compatible component. The sneaky thing about running an STA component on MTA threads is that everything seems to be fine - until you have threads overlapping and data gets corrupted. This can be subtle because the data corruption often affects table and memory points so you may get mixed up data in requests. Don't do it!!! <s>

But holy cow - the MTA performance is quite drastic, more than doubling the raw EXE COM throughput and and doing at very low CPU rates to boot.

But the real shocker is the abysmal performance of STA and COM+ operation. If I switch the pool threads to use ApatmentMode.STA in the HttpHandler, the performance of this COM server goes completely into the toilet. dropping by nearly 90%. Not only is the performance abysmal, but while it's so slow it's also sucking the CPU to nearly 100%. Compared to the EXE servers this is horrible.

Running with COM+ and Apartmentstate MTA also didn't help. What's interesting is as I watch the CPU usage is that when using COM+ the CPU load switches to the COM+ manager (in the COM+ Out of Proc Application anyway), so it's clear the issue here is how the COM object is getting marshalled.

STA Thread Marshalling Gone Bad

Now I'm not quite sure why performance is so drastically bad in all of the DLL scenarios. The way my particular COM pool manager works is like this: It creates a pool of threads up front and loads one instance of a COM object on each of these threads. The threads then continue to run in the background. When a request comes in the main ASP.NET thread requests one of these worker threads to do its processing and the actual request processing transfers over to the thread that's running the COM object. The COM object does its thing calling returning the result back to ASP.NET which picks up the response, writes it and completes the request.

I suspect the issue is the thread marshalling related to the STA threads. Although each of the COM object is running on a separate thread it appears that the COM marshaller is marshalling all of the STA state to a single thread and because of it ends up swapping data madly in and out of the STA Apartment.

STA threading is a problem in ASP.NET in general. For example Web Services don't support AspCompat mode directly and it requires a hack to make STA components work better in Web Services. I thought I'd give this hack a try as well basically by subclassing from the ASP.NET Page class and implementing as an IHttpAsyncHandler that uses the ASPCOMPAT BeginProcessRequest and AspCompatEndProcessRequest methods:

public class WebConnectionHandler :  Page, IHttpAsyncHandler

{

    public IAsyncResult BeginProcessRequest(HttpContext context, AsyncCallback callback, object extraData)

    {

        return AspCompatBeginProcessRequest(context, callback, extraData);

    }

    public void EndProcessRequest(IAsyncResult result)

    {

        AspCompatEndProcessRequest(result);

    }

    /// <summary>

    /// Overrides Page.OnInit which is called as part of Async callbacks

    /// </summary>

    /// <param name="e"></param>     

    protected override void  OnInit(EventArgs e)

   {

       /// Call my base handler
       this.ProcessRequest(HttpContext.Current);            

       HttpContext.Current.ApplicationInstance.CompleteRequest();

   }

}

This indeed gives me an STA thread for ASP.NET and it works well for components that are created and destroyed on the main ASP.NET thread, but does nothing for my COM components sitting on a separate thread.

My handler also supports SingleInstance mode operation which just loads a server on the main thread, does its thing and then releases the server. And I can see the ASPCOMPAT functionality working for the 'raw' COM server which clocks in at 905 requests/sec on the ASPCOMPAT thread with a reasonable 50% CPU load. But with the full West Wind Web Connection server this load and unload approach doesn't do very well, because West Wind Web Connection Servers are pretty heavy on startup and my particular server loads up several SQL connections and sets up various lookup tables which invariably is slow and resource intensive. The server is designed to be cached and not be constantly reloaded - so the STA tweak above unfortunately is not of a lot of help. 

Soooo... this leaves me wondering. Why is the STA performance so horrible even though each component is on its own thread? But maybe more to the point - I wonder what else needs to happen in order to get decent performance of an STA COM object. What is ASPCOMPAT doing behind the covers? I tried to look at the code in the AspCompatApplicationStep class System.Web.Util with Reflector, but that code degrades quickly into unmanaged code. ASP.NET is hacking deep to do whatever they do to provide decent COM performance.

This is not a big issue for me actually because EXE servers work well and better even than with the ISAPI interface and the EXEs are the preferred interface anyway because of the managability that is lacking from DLL components which once loaded can never be entirely released without a AppPool restart. Nevertheless, I was hoping that with this update DLL servers might become a more viable option for West Wind Web Connection, but at the moment it doesn't look promising. <shrug>