Rick Strahl's Weblog  

Wind, waves, code and everything in between...
.NET • C# • Markdown • WPF • All Things Web
Contact   •   Articles   •   Products   •   Support   •   Advertise
Sponsored by:
West Wind WebSurge - Rest Client and Http Load Testing for Windows

STA Threading Issues in ASP.NET


:P
On this page:

I've been working on replacing an ISAPI module which is a connector piece to an Application Server interface that interfaces with Visual FoxPro. In this interface the backend is managing a pool of COM objects that process requests and return results.

Dealing with COM objects might seem pretty passee in the age of .NET but nevertheless there's still a lot of that going around and I'm quite involved with this due to my FoxPro background and FoxPro's one interface mechanism to .NET being through COM. Anyhow, I essentially rebuild this ISAPI component for West Wind Web Connection in .NET in rather short order with great success.

The  connector is basically an interface that feeds request data from the Web server to an application server backend (a FoxPro application) and it can pass data between the connector and the server app in several different ways. One of these ways is via COM and the interface can work either with InProc our OutOfProc servers.

Now I know a few people are probably rolling their eyes at the thought of out of proc servers. However due to the nature of the interface and the ability to fully manage and shut down out of proc servers, out of proc servers are actually the preferred mechanism.

One big worry about this new handler interface has been performance compared to the ISAPI interface. This was one of things where there's no easy way to know beforehand what performance would look like so it took actually getting the COM pool manager into place before there was any way to see what performance looks like.

I was surprised to see that performance was actually slightly better than with the ISAPI extension. There are a number of reasons for that - one of which is the redesign of the pool manager using a set of threads that constantly stay active with the objects loaded on them. With the live objects on them (unlike COM Interop in ASP or ASP.NET) there's no reload overhead and unlike my original ISAPI extension there's no thread marshalling taking place. The server calls simply are routed to an available server on a separate thread.

So far so good.

STA for InProc and Out of Proc Servers

West Wind Web Connection primarily uses Out of Proc servers for various reasons. The idea is that servers are cached and kept alive so the performance difference between a DLL server and an EXE is not that drastic especially given the fact that 99% of processing time is spent in the actual FoxPro server code rather than in the connector code.

So when I originally built the component I built and tested with EXE servers - and got very respectable performance numbers from the HttpHandler. I'm running IIS 7 on Vista here and it doesn't seem like there are any limits (contrary to what Microsoft says) on the server as I was able to run straight ASPX pages through stress testing with well over 2000 requests a second (without any caching).

Performance of the HttpHandler turned out to be somewhat better than the ISAPI extension. I ran two sets of tests: One for raw COM interface performance by creating essentially a simple Helloworld method at the COM entry point, and another that runs through the full West Wind Web Connection framework and returns a full HTML page. The following are requests per second averaged out over 3  1 minute runs (right - it's just for rough over the thumb performance estimates <s>).

EXE Server

  ISAPI HttpHandler
Raw Method 811 980
WWWC Method  101 104

There's a big difference between the Raw COM call and the West Wind Web Connection basic response, because there's quite a bit of processing that occurs inside of the FoxPro server to set up for the request. Any amount of Fox code will drag these numbers down from the low level ISAPI/HttpHandler code. However, getting over 100 requests a second is still quite impressive.

What is nice is that the HttpHandler is actually a bit faster, although I noticed that the ISAPI version is less CPU intensive - apparently ASP.NET incurs a bit more overhead just getting to the actual handler code. The EXE server doesn't care about STA or MTA - it works the same either way since it runs out of process  and so doesn't need to used shared apartment thread state. This is pretty damn good performance for calling an EXE server on a heavy object like a Visual FoxPro COM object.

Now I can take the same EXE server and recompile it as a DLL server and run these same tests again against the HttpHandler, and here's where things get very weird:

DLL Server w/HttpHandler

  MTA STA COM+ (Lib) COM+ (App)
Raw Method 1811 15 18 20
WWWC Method  161 8 10 11

The tests are basically set up with the COM Pool running either MTA threads or STA threads, and then running MTA threads with COM+ as either an Inproc Library or Out of Proc Application.

I ran  MTA mode for the DLL merely for raw perf comparison. MTA threads are not an option for an STA component (like VFP) as it can cause major corruption of the COM server. These tests didn't stress Visual FoxPro much and so there was no corruption but data access in particular will eventually show issues with things like misplaced record pointers and any public variable overlap. Not an option but I ran it here to see the difference in performance that one could expect with an MTA compatible component. The sneaky thing about running an STA component on MTA threads is that everything seems to be fine - until you have threads overlapping and data gets corrupted. This can be subtle because the data corruption often affects table and memory points so you may get mixed up data in requests. Don't do it!!! <s>

But holy cow - the MTA performance is quite drastic, more than doubling the raw EXE COM throughput and and doing at very low CPU rates to boot.

But the real shocker is the abysmal performance of STA and COM+ operation. If I switch the pool threads to use ApatmentMode.STA in the HttpHandler, the performance of this COM server goes completely into the toilet. dropping by nearly 90%. Not only is the performance abysmal, but while it's so slow it's also sucking the CPU to nearly 100%. Compared to the EXE servers this is horrible.

Running with COM+ and Apartmentstate MTA also didn't help. What's interesting is as I watch the CPU usage is that when using COM+ the CPU load switches to the COM+ manager (in the COM+ Out of Proc Application anyway), so it's clear the issue here is how the COM object is getting marshalled.

STA Thread Marshalling Gone Bad

Now I'm not quite sure why performance is so drastically bad in all of the DLL scenarios. The way my particular COM pool manager works is like this: It creates a pool of threads up front and loads one instance of a COM object on each of these threads. The threads then continue to run in the background. When a request comes in the main ASP.NET thread requests one of these worker threads to do its processing and the actual request processing transfers over to the thread that's running the COM object. The COM object does its thing calling returning the result back to ASP.NET which picks up the response, writes it and completes the request.

I suspect the issue is the thread marshalling related to the STA threads. Although each of the COM object is running on a separate thread it appears that the COM marshaller is marshalling all of the STA state to a single thread and because of it ends up swapping data madly in and out of the STA Apartment.

STA threading is a problem in ASP.NET in general. For example Web Services don't support AspCompat mode directly and it requires a hack to make STA components work better in Web Services. I thought I'd give this hack a try as well basically by subclassing from the ASP.NET Page class and implementing as an IHttpAsyncHandler that uses the ASPCOMPAT BeginProcessRequest and AspCompatEndProcessRequest methods:

public class WebConnectionHandlerPage, IHttpAsyncHandler
{
 
    public IAsyncResult BeginProcessRequest(HttpContext context, AsyncCallback callback, object extraData)
    {
        return AspCompatBeginProcessRequest(context, callback, extraData);
    }
 
    public void EndProcessRequest(IAsyncResult result)
    {
        AspCompatEndProcessRequest(result);
    }
 
    /// <summary>
    /// Overrides Page.OnInit which is called as part of Async callbacks
    /// </summary>
    /// <param name="e"></param>     
    protected override void  OnInit(EventArgs e)
   {
       /// Call my base handler
       this.ProcessRequest(HttpContext.Current);           
       HttpContext.Current.ApplicationInstance.CompleteRequest();
   }
}

This indeed gives me an STA thread for ASP.NET and it works well for components that are created and destroyed on the main ASP.NET thread, but does nothing for my COM components sitting on a separate thread.

My handler also supports SingleInstance mode operation which just loads a server on the main thread, does its thing and then releases the server. And I can see the ASPCOMPAT functionality working for the 'raw' COM server which clocks in at 905 requests/sec on the ASPCOMPAT thread with a reasonable 50% CPU load. But with the full West Wind Web Connection server this load and unload approach doesn't do very well, because West Wind Web Connection Servers are pretty heavy on startup and my particular server loads up several SQL connections and sets up various lookup tables which invariably is slow and resource intensive. The server is designed to be cached and not be constantly reloaded - so the STA tweak above unfortunately is not of a lot of help. 

Soooo... this leaves me wondering. Why is the STA performance so horrible even though each component is on its own thread? But maybe more to the point - I wonder what else needs to happen in order to get decent performance of an STA COM object. What is ASPCOMPAT doing behind the covers? I tried to look at the code in the AspCompatApplicationStep class System.Web.Util with Reflector, but that code degrades quickly into unmanaged code. ASP.NET is hacking deep to do whatever they do to provide decent COM performance.

This is not a big issue for me actually because EXE servers work well and better even than with the ISAPI interface and the EXEs are the preferred interface anyway because of the managability that is lacking from DLL components which once loaded can never be entirely released without a AppPool restart. Nevertheless, I was hoping that with this update DLL servers might become a more viable option for West Wind Web Connection, but at the moment it doesn't look promising. <shrug>

Posted in ASP.NET  COM  Web Connection  

The Voices of Reason


 

Scott Walker
June 15, 2007

# re: STA Threading Issues in ASP.NET

I thought this link may be of some help to you. It certainly helped us understand what was going on with apartments and interop in one of our applications.

http://blogs.msdn.com/cbrumme/archive/2004/02/02/66219.aspx

Rick Strahl
June 15, 2007

# re: STA Threading Issues in ASP.NET

Scott, great article link - a little over my head in a few places. But it actually answered a few unrelated questions I've had.

From reading I get the feeling that this issue can't be addressed through raw .NET threads. It sound like the only way to get around this issue is as ASPCOMPAT does and offload the threads to the COM thread pool which is beyond my skill level (and not worth it in this app - given that Out of Proc has been working reliably).

However, I was really surprised to see that COM+ didn't work either in this scenario even in Out of Proc. I suspect that's because COM+ actually tries to reload the objects and there's the issue of CPU usage for startup being so heavy.

What's really needed for this to work is to be able to create multiple STAs and pin each thread to its own STA that's not shared. Since these objects only execute on the same thread there should be no marshalling issues, but right now .NET is pushing them all into the same STA and that's likely what's causing the problem.

E.R. Gilmore
June 15, 2007

# re: STA Threading Issues in ASP.NET

Greetings Rick,

We're dealing with that issue here. A co-worker sugested this article for more info on STA.

http://msdn.microsoft.com/msdnmag/issues/06/10/WickedCode/

Regards,

E.R.

Rick Strahl
June 15, 2007

# re: STA Threading Issues in ASP.NET

Yeah I know about that and I mentioned it at the bottom of the post. That works fine for anything fired on the ASP.NET thread, but not for code running on separate threads (or for components that are kept alive for that matter).

Kevin Dente
June 17, 2007

# re: STA Threading Issues in ASP.NET

It sounds suspiciously like your custom thread pool threads aren't running as STAs (in which case all the STA objects are marshalled to one thread). Are you sure they are running as STA? Did you check the native thread IDs in the managed code and the native code and see if they match?

Rick Strahl
June 17, 2007

# re: STA Threading Issues in ASP.NET

Yes all the threads on which the COM objects are running on are created with ApartmentState.STA. The thread is created then the COM objects (one for each STA thread) are loaded on it, then later requests are routed to these threads. The threads basically get signaled with WaitHandles and resume processing when a request comes in.

There's a definite difference in behavior if I switch between the two states - if I run with MTA threads I get a huge number of requests through but as I mentioned there's corruption of global data in the COM objects. If I run STA threads it *seems* like all requests are routing through a single thread although if I query thread ID in the actual COM objects themselves and echo it as part of the Response I get thread ids that change (ie. the threadIds of each of the threads in the pool).

I wonder though - there's some state passed from the main ASP.NET thread and the HttpHandler to the worker pool threads, which the worker threads use to retrieve request information (ie. ASP.NET's Context object), but none of that state is in any way passed to the COM objects... I wonder if there's some issue there. Probably not though because this all works fine with the Out of Proc components which don't have a problem and don't seem to care at all about MTA/STA (makes no difference) with the same performance regardless of whether MTA or STA threads are used for the pool.

Tomer
November 08, 2009

# re: STA Threading Issues in ASP.NET


I wonder did you implement a message pump in those thread?

Thanks

West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2024