Since the beginning of the year I've been working with 6 different customers on Web Service integration projects. Last year I went through a large number of customers along the same lines. It's not something that I enjoy doing either, but somehow I seem to get involved with these types of jobs quite  frequently <g>.

Now, we live in a world of Web Services and WCF in .NET which is in my opinion an awesome tool for both creating and consuming Web Services and it's been a tremendous help in interfacing with various services. But I'm also finding out that there are still a lot of services and SOA architectures that are rather difficult to interface with. The reality is that Web Services interoperability is far from ideal even today.

If you're building internal Web Services solution and you have control over both ends of the connection there are a lots of choices available and most likely you can build true end to end and easy to use solutions. However, it seems as soon as you're talking about external services hosted by various third party providers the story tends to look very differently. All of the 6 customers I worked with this month for example, were dealing with services that would not directly interface with .NET solutions and required extensive twiddling with schemas, custom WSDL files and marking up generated proxies by hand.

Not surprisingly all of the problem services accessed are non-.NET based and a few of them simply weren't standards compliant (even though they were based on well-known 'service platforms'). I'm even dealing with a heavy use third party service at the moment that doesn't support a unified WSDL schema - it uses SOAP messages and even WS-* message semantics but the service publishers only publish .xsd schemas and no WSDL. Unbelievable.

In addition some of the services I've been dealing with are based on Message Oriented principles clearly devised by people living in the Twilight Zone. One has a top level message object with 20 sub-level message objects and literally 200 related message types associated with this single message structure. The WSDL file for this single message structure is nearly 300k in size. One service end point, one message to handle what amounts to 20 different operations. The same architecture (again based on a widely used service platform) requires that messaging be provided by passing messages to another Web Service. The service itself doesn't return anything - not even a failure message, but requires that a message be sent to another service hosted at the clients. To top it off the server returns an invalid Http response (it returns no content body but has a content-length of 20 bytes). Yet this service's functionality is essentially a store and forward mechanism that could have been easily solved with a central hub that provides both message input and pickup. Instead customers (who happen to REALLY need the services features) are forced into complex installations and configuration of public facing Web Servers and are dealing with messages that are extremly unwieldy to work with.

I shudder to think at the amount of time that is collectively wasted on a badly designed service like this. Think about it - I spent around 25 hours figuring out the fixes to make the service work, building a receiving service along with a mechanism for storing the messages so that a client (desktop) application can get a freaking status update. That was 25 hours of my time along with probably a few hours bugging the service host tech people. Now repeat the same process for a few hundred customers. Add to that the developers at my client's who'll have to deal with these monster messages that are going on a 100k per message. And then you have the privilege of paying this company a few thousand dollars a year for using their powerful service based application. Aaaaargh...

I'm not complaining - it pays my bills and honestly to some degree I enjoy the problem solving inherent to this kind of work although digging through the 100 page service description manuals that are usually involved with these kind of apps often make me get a bit of vertigo: "Why the heck did I take this on again?"

No doubt there are also good services. If services are easy to work with I probably won't hear from anybody <s>. I suspect for every service that's problematic there are also many more that are in fact standards compliant, usable and provide good value for their consumers. But it seems there's an awful lot of crap out there from seemingly big iron and reliable vendors/organizations where you would simply think a little more effort could be expended to do it right. Not sure if it's just my bad luck that I've run into so many of these types of services recently or whether this is really a common scenario.

Web Service integration seems to crop up a lot in corporate applications these days - there's more and more of it and as much as we now have some extremely powerful tools to take advantage of them in .NET with WCF, the tools are not exactly easy to use. This is not really a fault of WCF itself - it's due to the inherent complexity in Web Service technology and represents in WCF with the number of options that are exposed and *can* be changed. The difficult part is discovering which options apply and when. I find the biggest problem here is that there's no linear way to learn about this type of stuff either. The only way you get familiar with it is through continued use and running into the edge cases. Most of this shit is so arcane that as soon as I figure it out I forget it again <g>. But at least next time I run into it I at least have a code sample to look into. Keeping notes on this stuff while working through problems with services has proven to be extremely useful as I can look up what I had previously done.

There are those in the Web Service space that preach contract first and 'angle brackets ueber alles', but I find there is a big disconnect between that and what most people that are tasked with using Web Services are familiar and comfortable with. While I can appreciate the immense knowledge that folks like Aaron Skonnard (who is brilliant and has helped me immensely with his articles over the years) have in the XML space, being familiar with every little nuance of XML and service standards is just not in the cards for most developers. Keeping up with XML standards is a full time career.I can't help and feel that Contract first and using WSDL to design your messages is akin to using Assembler to build a business application. It's difficult enough to keep up with the arcane details required for specific applications.

For example, I spent a few hours yesterday trying to digitally sign an XML document. How hard could it be to sign a section of  document with a certificate? It turns out it's not that difficult since .NET 3.0 introduced a number of helper classes in this respect and simple cases can be done with 20 or so lines of code. But then the Web Service options kick in. Which signature protocol is in use, which Uri canaconization, which encryption algorithm, which transform is applied to the signature and so on. Those 20 lines of code quickly went to 50 and most of the options to apply weren't documented anywhere and I found myself deeply mired in options I have never heard of and have even less of an idea what they do. At this point I'm just trying to match the Xml output to the sample I'm looking at.

A lot of credit to .NET in this case actually: I was happily surprised that I could get the signature completely right, but it took a good 3 hours to do something that simply shouldn't be this difficult. Why do we have to have 100 different combinations on how to sign an XML document based on Web Service Standards? Isn't the point of standards to simplify and agree on a common way of doing things? No you have 20 different options in combination that make things extremely complicated. WSDL in part solves this problem, but ah - in this case there was no WSDL since the vendor didn't deem it necessary...

It's not just other services that can be problematic though. WCF too can be a problem. Although standards compliant, WCF uses a distributed/fragmented WSDL schema when it publishes service WSDL meaning that the WSDL is split across multiple files. There's no built in option to prevent this from happening and it breaks a number of older Web service clients including the SOAP Toolkit (which I've run into with some older applications myself). You can bet there's some Java programmer somewhere using an older SOAP library cursing WCF in the same way I am cursing some of the Java service API that publish WSDL that WCF doesn't understand.

It works both ways. In the case of the segmented WSDL files it would have been nice if WCF at least would have provided a legacy option, but no - standards compliance overruled common sense and better interoperability. Luckily there's a way around this particular problem writing some code and a custom service factory as pointed out in the previous link but still this kind of arrogance is just the kind of thing that should be avoided at all costs. When it comes to Web Services the most important goal should be interoperability because that's ultimately what defines this technology.

We're not there yet...

So how about you? Are you finding many problem services as much as I do? Or is it just Murphy stalking me as usual <g>...