Rick Strahl's Web Log

Wind, waves, code and everything in between...
ASP.NET • C# • HTML5 • JavaScript • AngularJs
Contact   •   Articles   •   Products   •   Support   •   Search
Ad-free experience sponsored by:
ASPOSE - the market leader of .NET and Java APIs for file formats – natively work with DOCX, XLSX, PPT, PDF, images and more

Web Service Interoperability? Not from where I sit...


:P
On this page:

Since the beginning of the year I've been working with 6 different customers on Web Service integration projects. Last year I went through a large number of customers along the same lines. It's not something that I enjoy doing either, but somehow I seem to get involved with these types of jobs quite  frequently <g>.

Now, we live in a world of Web Services and WCF in .NET which is in my opinion an awesome tool for both creating and consuming Web Services and it's been a tremendous help in interfacing with various services. But I'm also finding out that there are still a lot of services and SOA architectures that are rather difficult to interface with. The reality is that Web Services interoperability is far from ideal even today.

If you're building internal Web Services solution and you have control over both ends of the connection there are a lots of choices available and most likely you can build true end to end and easy to use solutions. However, it seems as soon as you're talking about external services hosted by various third party providers the story tends to look very differently. All of the 6 customers I worked with this month for example, were dealing with services that would not directly interface with .NET solutions and required extensive twiddling with schemas, custom WSDL files and marking up generated proxies by hand.

Not surprisingly all of the problem services accessed are non-.NET based and a few of them simply weren't standards compliant (even though they were based on well-known 'service platforms'). I'm even dealing with a heavy use third party service at the moment that doesn't support a unified WSDL schema - it uses SOAP messages and even WS-* message semantics but the service publishers only publish .xsd schemas and no WSDL. Unbelievable.

In addition some of the services I've been dealing with are based on Message Oriented principles clearly devised by people living in the Twilight Zone. One has a top level message object with 20 sub-level message objects and literally 200 related message types associated with this single message structure. The WSDL file for this single message structure is nearly 300k in size. One service end point, one message to handle what amounts to 20 different operations. The same architecture (again based on a widely used service platform) requires that messaging be provided by passing messages to another Web Service. The service itself doesn't return anything - not even a failure message, but requires that a message be sent to another service hosted at the clients. To top it off the server returns an invalid Http response (it returns no content body but has a content-length of 20 bytes). Yet this service's functionality is essentially a store and forward mechanism that could have been easily solved with a central hub that provides both message input and pickup. Instead customers (who happen to REALLY need the services features) are forced into complex installations and configuration of public facing Web Servers and are dealing with messages that are extremly unwieldy to work with.

I shudder to think at the amount of time that is collectively wasted on a badly designed service like this. Think about it - I spent around 25 hours figuring out the fixes to make the service work, building a receiving service along with a mechanism for storing the messages so that a client (desktop) application can get a freaking status update. That was 25 hours of my time along with probably a few hours bugging the service host tech people. Now repeat the same process for a few hundred customers. Add to that the developers at my client's who'll have to deal with these monster messages that are going on a 100k per message. And then you have the privilege of paying this company a few thousand dollars a year for using their powerful service based application. Aaaaargh...

I'm not complaining - it pays my bills and honestly to some degree I enjoy the problem solving inherent to this kind of work although digging through the 100 page service description manuals that are usually involved with these kind of apps often make me get a bit of vertigo: "Why the heck did I take this on again?"

No doubt there are also good services. If services are easy to work with I probably won't hear from anybody <s>. I suspect for every service that's problematic there are also many more that are in fact standards compliant, usable and provide good value for their consumers. But it seems there's an awful lot of crap out there from seemingly big iron and reliable vendors/organizations where you would simply think a little more effort could be expended to do it right. Not sure if it's just my bad luck that I've run into so many of these types of services recently or whether this is really a common scenario.

Web Service integration seems to crop up a lot in corporate applications these days - there's more and more of it and as much as we now have some extremely powerful tools to take advantage of them in .NET with WCF, the tools are not exactly easy to use. This is not really a fault of WCF itself - it's due to the inherent complexity in Web Service technology and represents in WCF with the number of options that are exposed and *can* be changed. The difficult part is discovering which options apply and when. I find the biggest problem here is that there's no linear way to learn about this type of stuff either. The only way you get familiar with it is through continued use and running into the edge cases. Most of this shit is so arcane that as soon as I figure it out I forget it again <g>. But at least next time I run into it I at least have a code sample to look into. Keeping notes on this stuff while working through problems with services has proven to be extremely useful as I can look up what I had previously done.

There are those in the Web Service space that preach contract first and 'angle brackets ueber alles', but I find there is a big disconnect between that and what most people that are tasked with using Web Services are familiar and comfortable with. While I can appreciate the immense knowledge that folks like Aaron Skonnard (who is brilliant and has helped me immensely with his articles over the years) have in the XML space, being familiar with every little nuance of XML and service standards is just not in the cards for most developers. Keeping up with XML standards is a full time career.I can't help and feel that Contract first and using WSDL to design your messages is akin to using Assembler to build a business application. It's difficult enough to keep up with the arcane details required for specific applications.

For example, I spent a few hours yesterday trying to digitally sign an XML document. How hard could it be to sign a section of  document with a certificate? It turns out it's not that difficult since .NET 3.0 introduced a number of helper classes in this respect and simple cases can be done with 20 or so lines of code. But then the Web Service options kick in. Which signature protocol is in use, which Uri canaconization, which encryption algorithm, which transform is applied to the signature and so on. Those 20 lines of code quickly went to 50 and most of the options to apply weren't documented anywhere and I found myself deeply mired in options I have never heard of and have even less of an idea what they do. At this point I'm just trying to match the Xml output to the sample I'm looking at.

A lot of credit to .NET in this case actually: I was happily surprised that I could get the signature completely right, but it took a good 3 hours to do something that simply shouldn't be this difficult. Why do we have to have 100 different combinations on how to sign an XML document based on Web Service Standards? Isn't the point of standards to simplify and agree on a common way of doing things? No you have 20 different options in combination that make things extremely complicated. WSDL in part solves this problem, but ah - in this case there was no WSDL since the vendor didn't deem it necessary...

It's not just other services that can be problematic though. WCF too can be a problem. Although standards compliant, WCF uses a distributed/fragmented WSDL schema when it publishes service WSDL meaning that the WSDL is split across multiple files. There's no built in option to prevent this from happening and it breaks a number of older Web service clients including the SOAP Toolkit (which I've run into with some older applications myself). You can bet there's some Java programmer somewhere using an older SOAP library cursing WCF in the same way I am cursing some of the Java service API that publish WSDL that WCF doesn't understand.

It works both ways. In the case of the segmented WSDL files it would have been nice if WCF at least would have provided a legacy option, but no - standards compliance overruled common sense and better interoperability. Luckily there's a way around this particular problem writing some code and a custom service factory as pointed out in the previous link but still this kind of arrogance is just the kind of thing that should be avoided at all costs. When it comes to Web Services the most important goal should be interoperability because that's ultimately what defines this technology.

We're not there yet...

So how about you? Are you finding many problem services as much as I do? Or is it just Murphy stalking me as usual <g>...

Posted in Web Services  

The Voices of Reason


 

Mårten Törnquist
February 01, 2008

# re: Web Service Interoperability? Not from where I sit...

You should really have a look at the REST technology for web services. Once you start using it, you will probably never go back to SOAP. Everything is so much easier...

Max Christian
February 01, 2008

# re: Web Service Interoperability? Not from where I sit...

Yes yes and yes...

I think you hit the nail on the head with the word "arrogance". In real situations commercial realities affect everything and the company that has the upper hand commercially has a tendency to express that by doing whatever the heck it likes. Big firms spring to mind, but actually my own little companies have been guilty of this too in sometimes forcing partners to use techniques that aren't quite appropriate to the job at hand just because it's what we had "on the shelf".

Rick Strahl
February 01, 2008

# re: Web Service Interoperability? Not from where I sit...

Marten - I actually was going to write about that but put that off for another day. REST is really becoming the broad public API filling the void that Web Services complexity has left behind and maybe more so than Web Service providing the 'data at your fingertips'. Most public services that publish data (ie. Flickr, Facebook, Amazon (although Amazon also does WS) and Google) use REST based interfaces and for good reason since it's easier to use.

I have an article coming up for the WCF REST services. WCF REST services are workable but a big compromise between what WCF does and what REST does well and succeeding really at neither of them very well IMHO.

But REST is not adequate in all scenarios anyway. If security and and really complex messages are involved I think the REST interfaces fall pretty flat. And it doesn't solve the fat client scenario easily either - how would you effectively call a complex REST service from a fat client given that there's no REST client other than AJAX that really supports it?

Yaron Naveh
February 01, 2008

# re: Web Service Interoperability? Not from where I sit...

Rick
I work a lot with web services and you are right that there are a lot of wsdl/soap/whatever issues. But I think that the correct comparison is between "new" soap stacks from different vendors (e.g. wcf, axis2) and themselves. It seems that you are trying to build a client utilizing a relatively old technology and access a service utilizing a new technology (wcf). It is natural to have gaps between these technologies. You should look at the broader picture: Is the web services world going towards a convergence? The items to consider are:

1. Design time interoperability (mainly wsdl). Here I think we are in a good position with wsdl 1.1 and I see less and less problems with it. The two issues to be worried about are: (a) wsdl 2.0 which is not supported in wcf but is supported in some of the other platforms. (b) implementation-specific extensions to the wsdl which are common in the Java world.

2. Run time interoperability which divides into:

a. Xml/soap issues. Here we are in an excellent position since document/literal services tend to reduce the "web service interop" issue to "xml parsers interop" issue and this area is pretty safe.

b. New ws-* standards. This is the hardest part since these are new standards and their implementations are not grown enough. Additionally several vendors have took the freedom to extend these standards with proprietary protocols. Wcf is a good example since the default settings of a new service are not interoperable by default.

To sum up: By sticking to good old ssl and only basic ws-* standards one can make new soap stacks communicate successfully. Of course we could enforce WS-I on our services even before that but today WS-I compliance services are the out-of-the-box configuration of most soap stacks.

Yaron

Arun
February 01, 2008

# re: Web Service Interoperability? Not from where I sit...

Have you looked at Metro - the Web services stack in GlassFish ? We've done extensive interop testing with /NET 3.0 framework as part of Microsoft plugfests and fairly confident of interop with WS-* in .NET 3.0. Have a look at a sample demo that we showed last year:

http://blogs.sun.com/arungupta/entry/excel_using_wsit_javaone_2007

It shows how a secure and reliable Web service deployed on GlassFish can be invoked by Excel 2007 using .NET 3.0. No WSDL tweaking in this case. You can find more details on http://metro.dev.java.net.

Steve from Pleasant Hill
February 01, 2008

# re: Web Service Interoperability? Not from where I sit...

Even having a WS that is Java based on the server (with strict standards enforcement) and the client side is .NET caused us about 7 times the work over having both the client and the server piece in .NET. I guess that's just how it is.

"Standards" have different meanings and shades of compliance.

Don Demsak
February 01, 2008

# re: Web Service Interoperability? Not from where I sit...

Yes, I agree 100% And no, standards don't help (just look at how different the various browsers handle HTML+CSS).

As bad as some of the "other" platforms can be, Microsoft isn't immune. If you ever want a kick in the nuts, try calling the Sharepoint "Web Services" (they are web services in name only, basically POX over HTTP), and MOSS is suppose to be one of the flagship Microsoft server applications. And no, there are no WCF endpoints for MOSS, either.

Rick Strahl
February 01, 2008

# re: Web Service Interoperability? Not from where I sit...

@Yaron - sure if you look at the latest tools for everything things might look much better, but the reality is that most *server* implementations don't follow the latest tools and there's nothing you can really do to get around it if a vendor decides "Here's what we got, now you live with it." Basically that's the attitude I've seen in several of the projects I've mentioned.

OTOH I understand where vendors are coming from too. It's not easy for vendors that have services to move forward either. While it may make first time consumption easier, updating their services to a newer platform will likely break existing code (true even if you update say ASMX Service to WCF) or require maintenance of multiple side by side solutions neither of which is a great solution.

Frequent changes in standards just accelerate this process as versions and support infrastructure gets out of sync with the tools and existing implementations.

@Steve - no doubt have a homogeneous environment is easier because it's designed that way. I'm sure Microsoft works hard to build end to end solutions that work on the MS platform. Unfortunately, in my experience at least, the vast majority of public vendor services I've worked with are not built with Microsoft technology. Not all of it blows completely as seems to be the case right now with these recent customers I'm working with - I've also seen a few very complex services that "just worked" which is almost a surprise when it happens <g>...

Randy Jean
February 01, 2008

# re: Web Service Interoperability? Not from where I sit...

Wow, you are preaching to the choir. I was beginning to think I was the only one that struggled with the various implementations of web services. I do mostly consuming (in VFP and .NET) but have started to publish as well and it's amazing how easy .NET makes it to create a robust WS. One of the first integration projects I did was SOAP without the ROPE (ie. no wsdl) it was non-MS and I think it was an Apache service of some type - it sucked, but fortunately the vendor listened to complaints and switched over to .NET services. If I didn't have the wwsoap class it would have been even worse. Every project since has been something different. What I really like is the package software vendors who change their standards and interfaces on every update without any notification. For instance, one package (I won't mention the name) switched from returning exceptions as string messages to throwing soap exceptions for things like "vendor not found" - wasn't a big deal to handle with some TRY/CATCH but in the meantime it broke things very badly - fortunately we hadn't gone into production yet.

Still, even with all the negatives it still beats working with clients that want to do all their integration in batch mode passing Excel files back and forth over FTP - seems like the mortgage industry including some very large companies still like to do things this way. Ugh!

Anyway, just glad I'm not the only one. I really enjoy reading your blog Rick. It's a great contribution to the development community.

- Randy

Steven Black
February 01, 2008

# re: Web Service Interoperability? Not from where I sit...

>> At this point I'm just trying to match the Xml output to the sample I'm looking at.

Is it just me or do folks find that, in the end, nothing ever beats this?

I can't count the times wheel-spinning stopped shortly after some real-world XML inputs or outputs finally appeared...

Rick Strahl
February 02, 2008

# re: Web Service Interoperability? Not from where I sit...

> Is it just me or do folks find that, in the end, nothing ever beats this?

@Steve - if you take the pragmatic approach that's definitely the end all. Hopefully though somebody had the foresight to create a sample you can build of because usually the documentation seriously lacks in this regard.

Part of the point is that it shouldn't have to be that way, right? That's what standards and toolsets are supposed to solve for us so that I don't have to look at an XML document and figure out the 100 variations of how I can do the digital signature for the XML document, but rather have the platform discover it and then provide it transparently when I call the service and maybe pass it nothing more than the certificate to use.

WCF actually can do this, but I think we're still quite a ways away from a point where that information is consistently available or even having the tools that can work with this kind of rich schema information.

Imagine for example, trying to do this with VFP and the SOAP Toolkit which would be a real barrel of laughs, eh?

shudder to think
February 02, 2008

# Web Service Interoperability? Not from where I sit...

Bookmarked your post over at Blog Bookmarker.com!

Ryan Haney
February 04, 2008

# re: Web Service Interoperability? Not from where I sit...

I agree all the way. Every once in a while we have an INTERNAL SOAP issue where our Java team can't parse the WSDL from our .NET web service. You know what we resort to 99% of the time? A simple request that accepts a string and looks a query string! So much for all that "cool" XML.

I believe REST style services is really the way to go. I just finished the first draft of a payment service that was coded in a REST style. It supports subscriptions that are associated with memberships and transactions. Creating a subscription attempts to create the membership, and processes the transaction. I use REST style urls like /subscriptions, /subscriptions/1, /transactions, /transactions/1, /members, /members/1, etc. with the standard HTTP methods (GET, DELETE, POST, PUT).

The entire time I was developing this app, I have been abstracting out framework code so I can reuse these concepts in other apps. The basic object flow is this:

HTTPHandler->Converter->Resource[->Optional Template->Form]

The HTTPHandler acts as a Front Controller and is used to abstract the HTTP protocol details from the Resources. The HTTPHandler also takes some HTTP headers into account and maps a Converter, whose job is to translate the request into a generic request format. This generic format is then used by the Resources to validate, process and return a response, be it data or information / error messages.

The Converter component, in theory, could allow many different types of requests and responses to be processed - so if a client chooses to send a JSON requests and receive XML responses, they can. Resources currently use arrays (might refactor to classes, I am coding this solution in PHP) but could easily use some well known base class or reflection with attributes in C# - the Converter just needs to know how to translate the data received. Resources decide if authentication is needed via a boolean property, and let the HTTPHandler pass credentials inside the standard HTTP Authenticate header using Basic authentication. Since we are dealing with sensitive data, I am hosting it over SSL for encryption. Resources are also extensible. I currently have a SimpleDatabaseResource which by overriding some very basic methods provides CRUD functionality. Since the web service uses standard HTTP, I can use a browser to interact with the web service - technically the only tool I need, and I don't need to code much more than an HTML form. However, if I chose to, I could create a client in C#, Java, Perl or any other language that supports communicating with HTTP using custom headers. If I wanted to get really fancy, I could build objects that would make that the communication over HTTP fairly transparent.

I will admit though, that wrapping your head around the REST style is a slight learning curve. Once you get it though, the elegance shines through.

As a side note, once my implementation of REST is proven using this application and I have had a chance to refactor the code a little, I plan on releasing it under the GPL in PHP and C# flavors. In fact, I have already registered PHPReST.com and CSharpReST.com. There isn't any content up there yet, but will be soon.

Francis Carden
February 07, 2008

# re: Web Service Interoperability? Not from where I sit...

Rick, awesome blog and so very important to make people AWARE of this BEFORE/WHILST they embark on their SOA strategy. Interesting comments on REST.. Funny that simplicity wins over every time when it comes time to deliver..

Last point.. You said...

<b>And it doesn't solve the fat client scenario easily either - how would you effectively call a complex REST service from a fat client given that there's no REST client other than AJAX that really supports it? </b>

Take a look at www.OpenSpan.com that enables almost any FAT client (RIA/thin clients too) to consume web services (including REST). i.e. not just Ajax. What OpenSpan calls - <b>the last mile of SOA</b> - crucial to getting SOA deployed to business Right Now.

Rick Strahl
February 07, 2008

# re: Web Service Interoperability? Not from where I sit...

Actually I oversimplified the WCF REST Services a little in my remark <s>. You can do pure rest for non-Ajax code as well since those REST services allow angle bracket messages as well as JSON messages - although you can't very easily mix both from the same endpoints (which is pretty lame given WCF's protocoal transparency trackrecord previously).

REST works great, but if you end up with solutions that currently require WS* features, then REST becomes much less attractive because it would have to essentially re-invent that feature set over REST as well. Most of this would be authentication, encryption and signatures which seem to be the most used WS* features.

But for open services REST is already king of the hill. If you look at any public service like FLICKR, FaceBook and even Amazon (which has both SOAP and REST, with REST actually getting used more and more though) they all focus on REST for simplicity because it can be used from anywhere without the need of a sophisticated SOAP client. Most importantly browsers can consume it.

It also is pretty telling of complexity that after all these years browsers don't include some sort of native SOAP client. We should probably be glad for that, but imagine if there was a more generic SOAP client available in browsers Web Service's adoption rates would probably look much differently.

Ryan Haney
March 01, 2008

# re: Web Service Interoperability? Not from where I sit...

"Most of this would be authentication, encryption and signatures which seem to be the most used WS* features."

Actually, this, IMO, is what is most misunderstood about ReST. When HTTP is used to transport data for ReST services (and when is it not?), support for authentication and encryption are built in via the Authenticate header and SSL. There are many other benefits that come with the embracing the Internet, such as proxies/caching. Signatures wouldn't be that hard to implement given a properly designed object model, but why not use the standard methods of authentication over an encrypted communication channel?

"Most importantly browsers can consume it."

Browsers still aren't the perfect ReST client. They usually only support GET and POST methods. So what about PUT and DELETE?

"[...]how would you effectively call a complex REST service from a fat client given that there's no REST client other than AJAX that really supports it?"

Any fat client that supports communicating over HTTP can communicate with a ReST service. Take the System.Net.WebRequest class in .NET. For RIAs, the HttpService class in Adobe Flex.

Rick Strahl
March 01, 2008

# re: Web Service Interoperability? Not from where I sit...

@Ryan - there are really two issues here: Transport level services which is basically what REST provides and Message level services which is what WS* provides.

I won't argue the benefits of REST with you - I'm pretty much sold on using REST whenever possible simply because it's easier to work with typically on both ends of the connection. But you have to admit that there are plenty of Enterprise scenarios where pure REST over SSL and the limited HTTP based authentication isn't enough.

I tend to agree though that often times even those issues can be addressed and in fact even many Web Services that i've been dealing with still deal with interfaced based tokens for security which is easy enough to implement (albeit requires reinventing the wheel).

West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2020