Rick Strahl's Web Log

Wind, waves, code and everything in between...
ASP.NET • C# • HTML5 • JavaScript • AngularJs
Contact   •   Articles   •   Products   •   Support   •   Search
Ad-free experience sponsored by:
ASPOSE - the market leader of .NET and Java APIs for file formats – natively work with DOCX, XLSX, PPT, PDF, images and more

Multi-way Messaging - To Queue or not to Queue


:P
On this page:

 

I'm always been intrigued when I review official design guidelines that call for Message Queues or any Queue related technology for various messaging scenarios. Queues seem to be the preferred mechanism that you see in Service Oriented designs. For .NET there's MSMQ and there's Sql Server 2005's Sql Service Broker which is basically a queue based implementation that can be driven directly through SQL code.

 

I think it's great that there 'in the box' solutions to handle this for you, but I've never been able to make Message Queues fit my distributed scenarios. The problem with queues is, well that they are queues. They are optimized for first in first out operation. Which in most scenarios that I end up with just doesn't work, or at the very least works very badly. Surely there are scenarios where queues are a good fit, but I find that I rarely need that particular scenario. A FIFO scenario works good for very structured scenarios and deliver once messages, but it works very badly for ongoing and two way messaging that seems to be much more prevalent (at least for me).

 

In most of my messaging scenarios I do need a dedicated reliable message store that allows applications to communicate with each other, but typically they need to be able to be random access. Specifically in many scenarios there's two way messaging where one application submits a message, another application processes a message and the first eventually gets a result returned to it. Now you can do this with queues where you have two queues set up – one for each of the messagengers, but that's both inefficient and complicated to work with (Ok, not that complicated but certainly not logical).

 

Let me explain one scenario I constantly see. I've been working with a customer who needs to have a daemon type process that's responsible for running some heavy duty reports. These reports need to be processed on a separate machine because they are serious CPU hogs and simply don't work well on the Web Server (or other application server for that matter). Most of the reports take about a minute or two to run and are typically served over the Web with an ASP.NET front end.

 

So it's obviously too long to run these requests right inline of the ASP.NET Page and because of the CPU load we are in fact offloading the processing to another machine. In addition, the server application should fire back some progress information about what's going on because clients will wait for the result to return. The timing is such that it's just quick enough to let users wait for results.

 

The way I've built this sort of application is by essentially creating a messaging interface that allows the ASP.NET application to communicate with the Report Server. This particular implementation uses SQL Server. It stores messages in a SQL table – each request has an id and the report server and ASP.NET communicate basically through this single record in a SQL table. Each record has information about when it was started and the last update time, fields for several types of messages (a message code, and a text message, a completion percentage) and of course cancel and completion flags. It also has a binary image field that can be used to hold any results which can either be the actual result or some sort of message or pointer to the actual results. In some situations the results are shipped back as binary serialized data (such as a serialized object or DataSet) at other times it may point to a disk resource or it may be an XML Message with more information.

 

I posted a sample a while back with my ATLAS examples that demonstrates a simplified version of the message handler class here:

 

http://www.west-wind.com/presentations/scriptcallbacks/sample/Atlas/LongProgress.aspx

 

 

The sample is overly simplistic and actually processes the request in the context of the page (on a separate thread) but it uses this sort of messaging class to communicate back status information for the current operation.

 

It actually works well, but I keep wondering why THIS sort of scenario isn't built into the OS in some way. In fact when Serivce Broker came out I was hopeful because it was SQL based that it would support better random access stores that can be accessed by multiple clients and can retrieve message out of order. Maybe this is possible but from what I've looked at so far it looks like messages are even more strictly queue based than MSMQ.

 

I've built this sort of interface with queues before too. It can be done, but it's a heck of a lot more work and much more complicated because queues are not designed to be accessed randomly (although with MSMQ at least you can – MSMQ can also peek messages which SB can't as far as I can tell). In most two-way or multi-way messaging scenarios you need multiple queues, and if multiple clients are listening to response queues a lot of same data gets written to the same queue multiple times because you can read data only once.

 

To top it off MSMQ is not really designed for large messages. MSMQ has max Message size of 4MB – if you have messages any larger you have to split them up. 4MB seems large until you have a result you need to queue up. Splitting and recombining that message is royal pain.

 

So, here I am wondering if I'm the only one who's wondering if there isn't a more generic solution out there that addresses this sort of random access messaging, communication problem that results in constant redesigning of custom solutions.

 

And if not I guess I should publish my Messaging class <g>


The Voices of Reason


 

Doug Dodge
April 23, 2006

# re: Multi-way Messaging - To Queue or not to Queue

Rick, You are absolutely not the only one 'out there' who has needed this ability. Random access strikes me as being much more 'real life' than sequential, though they both have their place. DD

kevin
April 24, 2006

# re: Multi-way Messaging - To Queue or not to Queue


Rick, I'm working on a distributed application with the requirement for multiple communication services (email, faxing, sms, etc) and we're looking into handing off these types of communications to MSMQ for more effecient async processes. The communication services themselves will be exposed via a webservice, so all the peripheral apps can pass along an object, and based on the communication type (another arg passed to the webservice) a class factory loads the correct communication provider, executes an xslt transformation on the object which returns the desired communication formatting (html, pdf, txt, etc) and then executes the Send() - all this seems to be a lot of work, so we are looking into queue this actions so the initiating web application can move along with its processes.

any thoughts?

also, let's see that Messaging class :)

I'm gonna bounce around your site...I see lots of great atlas tutorials on here (great job) and I'm currently familiarizing myself with atlas, and I'm writing a lot of client script. I'm not a fan of xml script(yet). I'm sure I'll have some atlas questions for you if you don't mind.

thanks and keep up the great content.

Arthur
April 24, 2006

# re: Multi-way Messaging - To Queue or not to Queue

But if you can randomly access it, it isn't a queue any more, is it?

Rick Strahl
April 24, 2006

# re: Multi-way Messaging - To Queue or not to Queue

Arthur, right but that is my point <g>. Queues are a particular scenario that IMHO is getting abused into situations where Queues are not the right way to go.

Most services end up requiring two-way communication be it for process progress information or for communicating final results. As mentioned you can do this with queues but it's pretty messy to do so as you need to deal with multiple queues for each type of messages (since you can't easily check message types).

In most scenarios I work with there's sender addressing of some sort involved - another scenario not handled by queues. You have a generic 'queue' to hand messages off to and listeners can look at this queue not sequentially but to pull out the messages that it is listening for...

I bring this up primarily because I'ev been reading a number of papers on this recently that were using queues for the above purposes exactly and FIFO is a horrible fit.

Rick Strahl
April 24, 2006

# re: Multi-way Messaging - To Queue or not to Queue

Kevin, if your messages are one way and don't require progress information back to the original requestor, then Queues work great. If you can use MSMQ or SQL Service Broker without much of a hassle I would by all means do so because these architectures are designed for high reliability and redundancy. Any solution you home brew (like mine) takes some time to mature and find all of the little quirks that might go wrong.

What I have works very well actually for what I need it for, but it's not on the same level of security, stability as say MSMQ.

Rick Strahl
May 15, 2006

# re: Multi-way Messaging - To Queue or not to Queue

Several people have commented offline that what I'm asking for is not really a queue mechanism.

I actually beg to differ. Yes it's not a pure Queue design, but a hybrid. I want Queue semantics on the message submission and retrieval, so that message delivery is essentially FIFO.

But then, once the message has been picked up, both the client and the server should still be able to communicate somehow within the context of the message.

So my implementation basically use a SQL Record that acts as the 'Messenger' along with a .NET front end class that provides easy methods to write out common message semantics (CompleteMessage, CancelMessage etc.) that handle all the detail of managing the ongoing request.



West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2019