Rick Strahl's Web Log

Wind, waves, code and everything in between...
ASP.NET • C# • HTML5 • JavaScript • AngularJs
Contact   •   Articles   •   Products   •   Support   •   Search
Ad-free experience sponsored by:
ASPOSE - the market leader of .NET and Java APIs for file formats – natively work with DOCX, XLSX, PPT, PDF, images and more

Silverlight and Data Communications


On this page:

So I’m finally getting an opportunity to work with Silverlight in a real application. I had mentioned this application in the past a few weeks back – the original application architecture was created as a Flash front end to a .NET backend which provides service style XML data. In the original app the server infrastructure was used purely to serve data to the client – basically a complex, mostly meta data drive SQL Engine based on complex stored procedures feed the client Xml data. Originally our data format has been DataSets because those were the easiest way to consume the data in the Flash client since there’s native support for DataSets and it greatly facilitated the process of passing data to flash and the format is basic enough that it’s easy to send update data back from the client (manually created in this case).

In the original app I basically built the data engine – which is a custom HttpHandler based affair that basically is a glorified XML routing engine that maps inbound requests from the client who sends XML data to the appropriate business object/Stored Procedure routine which in turn responds with a data result that is streamed back to the client. The server app and business objects actually do very little work in this process – most of the work lives in the SQL backend with its stored procedures that manipulate the data. The business object’s tasks mostly is to pick up inbound query parameters and map those into the appropriate stored procedure calls. The results for now are returned were returned as DataSets and there’s a bunch of extra logic that can also return that same data out as either JSON or as simplified XML (without DS schemas).

This has all worked well, but it’s been based on the existing Flash expertise at the time, which have now become an issue.

So based on some back and forth discussions the thinking now is that since we’re starting from scratch Silverlight is probably a better choice due to the easier coding model and readily available developer pool. Which has landed me smack in the middle of Silverlight and not least of all a data access decision to make. :-}

Data Models

The main issue I’m working on at the moment is finding the right data model. The server side as it sits right now has a number of options for returning data to the client. Silverlight has many options for getting data to the client:

Raw Xml

Obviously we can retrieve the raw XML from the server and just parse that on the fly. While not difficult this would not be my choice for data access because it’s tedious and potentially error prone. Since the data is mostly dynamic and doesn’t lend itself to mapping well though this is not totally unreasonable because we’ll likely end up with some sort of untyped data representation on the client no matter what.

JSON or ADO.NET Data Services

As I mentioned our backend can actually feed data out as JSON so another option I was toying with is to use JSON marshalling to map onto objects. The problem with this approach is that we don’t really have a model we can map to. The problem is that the backend (over which I have no control at this point) is just a fairly large set of heavily parameterized stored procedures. While the actual simple tables in the system are fairly small and could be modelled, the way the data is manipulated and returned does not lend itself well to producing either a Linq to SQL or Entity Framework data model. We’d end up with about a hundred or more hand created types that would have to be kept in synch with the database. That doesn’t sound very promising either.

ADO.NET Data Services could be interesting for certain data scenarios, especially free form data access, but my current scenario isn’t one of them I think. Still digging through the docs on ADS but I think when dealing with stored procs there’s really nothing here to help effectively. Nor is the structure of this architecture really a good fit. To make this work would be square peg, round hole.

Web Services

Web Services provide another possible option. Silverlight can directly access WCF Web services and retrieve data that way and this is actually a pretty easy interface to work with. But again it requires fixed, typed result sets so the issues are somewhat similar as the previous  item. You need to be able to full type information published on the server in order to get the data back to the client.

One thing I was thinking about was to return a result back from a Web Service as a generic dictionary which actually serializes properly, preserving underlying type information:

[OperationContract]
public Dictionary<string, object> GetDictionary()
{
    Dictionary<string, object> items = new Dictionary<string, object>();

    items.Add("Field1", 10);
    items.Add("Field2", DateTime.Now);
    items.Add("Field3", "Rick");

    return items;
}

Properly returns in Silverlight as:

private void btnService_Click(object sender, RoutedEventArgs e)
{
    PraServiceClient service = this.GetService();
    service.GetDictionaryCompleted += 
        new EventHandler<GetDictionaryCompletedEventArgs>(service_GetDictionaryCompleted);
    service.GetDictionaryAsync();
}

void service_GetDictionaryCompleted(object sender, GetDictionaryCompletedEventArgs e)
{
    if (e.Error != null)
    {
        MessageBox.Show( e.Error.Message, "Service Error",MessageBoxButton.OK);
        return;
    }

    Dictionary<string, object> items = e.Result;

    StringBuilder sb = new StringBuilder();
    sb.Append(items["Field1"]);
    sb.Append(items["Field2"] + items["Field2"].GetType().ToString()) ;
    sb.Append(items["Field3"] + items["Field3"].GetType().ToString());

    MessageBox.Show(sb.ToString());
}
private PraServiceClient GetService()
{
    EndpointAddress addr = new EndpointAddress("http://rasnote/PraWebService/PraService.svc");
    BasicHttpBinding bind = new BasicHttpBinding();
    return new PraServiceClient(bind, addr);
}

The above surpringly works to marshal types in the dictionary down to the client even though the dictionary is Dictionary<string,object>. WCF does not like serializing generic object types however so results of type object or anonymous types fail outright. I suspect using these types as values for the dictionary too will fail.


Dynamic Data Access

The real issue here is that in lieu of a data model that can be used on the client side there’s no good data mechanism built into Silverlight that provides access to some dynamic data returned from the server. So say the server returns an arbitrary query where the user is allowed to pick the columns he wants to retrieve there’s really no clean way to return that data and present it on the client side. JSON serialization requires a type as an input to understand the structure and so a dynamic results can’t be represented. It’s possible to actually use JavaScript to create an object but even then that object comes back as a dynamic object which could only be accessed by Reflection (C# 4.0 dynamic oh where are you? <s>)

No DataSet Support

I know DataSets are kind of passé but when I really think about how this existing data engine works – DataSets with their weak typing and even with their update support are the perfect fit for this type of application. As many problems as DataSets have in typical pure data access scenarios, in internally controls disconnected scenarios Datasets have features that none of the newer data techologies have matched to date – mainly because DataSets allow for dynamic data to be returned and be accessed easily.

Alas – Silverlight has no support for DataSet/DataTable/DataRow. While looking at various third party components I actually ran into ComponentOne’s Silverlight Controls which include a Data assembly that provides a minimal set of the classic ADO.NET objects for Silverlight. But even with those the support is fairly limited (for example no support for schemaless XML or Diffgrams – the latter of which is kind of a bummer). Still for now that’s something to start with – if we decide to go this route I suppose it’s not terribly difficult to build my own abstraction that does support this functionality.

Something Custom?

I’ve also toyed with the idea of building a custom Dictionary type interface that I can deserialize from JSON which would basically allow me to use JSON from the server and get some sort of generic object back back on the client. Since the data is for the most part table based, object hierarchy isn’t really an issue either so it’d be fairly straight forward to build up something like this fairly easily.

Yes, C# 4.0 and DynamicObject would be pretty handy for this, but it’s going to be some time before that comes to pass.

Hmmm…

In some ways these issues are actually easier to work through in JavaScript/Html based interfaces. Because Javascript is a dynamic language it’s actually easier to retrieve JSON data from the server and use it on the client. The server can return arbitrary data and simply return it with the client having direct access to it. I suppose one could choose one of the dynamic languages (JavaScript, IronPython or IronRuby) on the client, but frankly that’s not really a good option either. The whole point of using Silverlight is to provide the richer, statically typed environment.

So I’m curious to hear what others are doing for their client side ‘data’ connectivity, especially when dealing with data that can’t be easily serialized back and forth between client and server. I suspect a lot of folks probably deal with raw data – manually parsed XML or possibly. If not what are you doing for data transfers? There are a many options but it seems that most of them have some shortcomings when it comes to effectively dealing with the data on both ends of the connection.

Posted in ADO.NET  Silverlight  

The Voices of Reason


 

pete w
November 20, 2008

# re: Silverlight and Data Communications

datasets? passe? thats silly to me, its still a perfectly useful tool in the shed. (some people use it like it is the ONLY tool in the shed, but that is another story =))

if you like datasets, and you can work with xml, then maybe you could capture the xml and then do something like:
DataSet ds = new DataSet();
ds.ReadXml(xmlString);

just a thought

Jim
November 20, 2008

# re: Silverlight and Data Communications

<em>So I’m finally getting an opportunity to work with Silverlight in a real application.</em>

You just summed up why I haven't jumped on the Silverlight bandwagon. Something like 80% of the computers out there don't even have Silverlight installed. What MS is doing with LINQ is not exactly inspiring confidence, either.

Matt
November 21, 2008

# re: Silverlight and Data Communications

Jim,

Do you mean "What MS is doing with LINQ-to-SQL" or something else?

Perry
November 21, 2008

# re: Silverlight and Data Communications

Arrgh!!! I hate it when I see system with complex stored procs! All that logic should be in the middle-tier.

I think this is a good time to evangelize the benefits of having the data orchestration in the middle-tier as opposed to the backend (SPROC). I suppose they have no automated testing as well.

My take on the whole situation is that you need to explain that this architecture might have been acceptable say 5 years ago, but today things need to be done right. Your data tier is just for CRUD operations and your middle-tier is for orchestrating (pulling in the right data and transforming it for the client). Then you are free to choose how to send it back to the client.

So what do I do in this situation? Convince the client to do it properly or I walk away. Fortunately I have the luxury of doing that.

pete w
November 21, 2008

# re: Silverlight and Data Communications

so, you think stored procedures should never be used?

Paul Mrozowski
November 21, 2008

# re: Silverlight and Data Communications

Are the results of the customizable queries just a possible subset of some know max. # of columns? For example, they can select 0 or more columns from data in columns A, B, C, D, E, F (however they're generated)? If so, you could just create a thin class wrapper for all of the available columns (and maybe hold a collection of the "used" columns). You'd probably want to automate building this kind of wrapper and make them get automatically generated as part of the build process. That would minimize most of the pain.

Otherwise a dictionary would probably be the easiest to implement.

David Betz
November 21, 2008

# re: Silverlight and Data Communications

I haven't used a DataSet or DataTable in at least 3 years. I also write *all* assemblies in WCF services and pass things are as either scalars, DTOs, or List<T> of DTOs. Because of this, all my services are reusable in my existing web sites and WPF apps and now in Silverlight.

As far as connectivity in Silverlight, see http://www.netfxharmonics.com/2008/11/Understanding-WCF-Services-in-Silverlight-2

pete w
November 21, 2008

# re: Silverlight and Data Communications

I think what rick is desiring is something more dynamically typed than custom DTOs and List<T> can offer.

The type checking is a nice practice, but it gets in the way when all you want is late binding.

If you find yourself heavily relying on reflection, if you try programming in boo or ruby you will see what I mean

Kevin
November 21, 2008

# re: Silverlight and Data Communications

Rick,

I am beginning to learn Silverlight 2 and one of the things I discovered was that many of my classes were incompatible with Silverlight because they were not created within the Silverlight CLR. For example, I use the Enterprise Library Data Access, Error Handling and Logging Blocks for many of my classes and it cannot be used, which means that I to have recreate those classes using the Silverlight Class Template, if I need to use those specific classes. Since you are starting from scratch, this probably will not be an issue. I am in favor of what Perry stated above, my Data Access Layer performs the CRUD functions and my Business Layer transforms that information as objects that are usable at the Presentation Layer. I try to follow the KISS principle as much as possible!

Another issue I ran into at work is that my clients reside on a closed network, they do not have access to the internet or admin privileges on their machines. They cannot download or install the needed Silverlight component. I see many applications where Silverlight could possibly be a good fit, however, I will not be able to implement any solution for some time because of the security of the network. Because of that, I use a JavaScript/HTML approach that you mentioned, which is why I really appreciated your presentations at Dev Connections.

Aloha,

Kevin

Rick Strahl
November 21, 2008

# re: Silverlight and Data Communications

In my own applications I'm also in favor of doing logic in the business layer - when you do that it's much easier to model the data and push the model to the client. No question this is desirable, but realistically it's not always possible even in the best of circumstances. if you have a dynamic system (say something that generates reports that aren't based on fixed fields) where do you get the fixed model from? This is a real problem and really Silverlight natively doesn't address this well at the moment because all data transfer mechanism (short of raw XML parsing) rely on static types.

@David - sure WCF is fine and typically I do the same - returning custom entities. As long as you have fixed types to work with that's all fine and well. The problem is if you don't as is the case here with the existing backend. I'm still looking at other options which are to try and generate all possible types on the backend but it'll be mostly a manual process of mapping queries to custom types which is a pain and maintenance heavy.

The SP approach is not my preference either but it's what the backend system brings out at the moment so I have to work around those issues. WCF is no help in this particular scenario because I can't return these dynamic query results - the WCF Serializers will choke on object results and anonymous types which potentially would allow working around this issue.

When using Web apps this is often not an issue because client side JavaScript can easily take the dynamic types that a JSON serializer spits back (not the WCF one, but JavaScript Serializer which DOES work with object and anonymous types as well as JSON.NET or my own Serializer).

Rick Strahl
November 21, 2008

# re: Silverlight and Data Communications

@Pete - when it's all said and done using DataSets is the easiest approach in this system because a) the data is already capable of generating DataSets and XML output from them and b) it's a dynamic data format. The problem is that Silverlight doesn't support them.

Stored Procedure results are OK - no issues with some store proc results especially those that are meant to be read-only to retrieve data. In this case though basically the whole system is wrapped up in stored procs which from the data end of things makes it difficult to build a model of any kind or even discover all the types used.

@David - while I can appreciate dislike for DataSets, in some situations they are just a good fit precisely because they are NOT strongly typed and because they are extremely easy to serialize between client and server.

I would like to hear how you'd work dynamic results with just WCF as you mention above... Just curious. Map every single result to hand created types? That's the only way I see.

Stephen
November 21, 2008

# re: Silverlight and Data Communications

if you can serialize as JSON from your data tier, then you could use the serializer from JsonFx.NET to deserialize. The resulting object graphs are all standard .NET collections so there would be no custom types to maintain. You could even use LINQ and anonymous types in your results since they come back as dictionaries, lists and primatives once deserialized.

Marc Arbesman
November 21, 2008

# re: Silverlight and Data Communications

This could be way off. But could you ask for the stored procs to be implemented in a view? You have more options. It seems obvious. So it's most likely not possible. But I thought I'd throw it out there.

Peter Bromberg
November 21, 2008

# re: Silverlight and Data Communications

Back before the DataSet supported binary serialization, MS's Ravinder Vuppula provided a surrogate that basically flattened out a DataSet into arraylists (now you would use List<t> etc.). Here's the old KB: http://support.microsoft.com/default.aspx?scid=kb%3ben-us%3b829740

What I've done with Silverlight is to use a variant on Greg Young's "Fast Binary Serialization" (hacked for Silverlight) combined with optional Zip compression. This all works on Silverlight and of course can be duplicated server-side. So what you get is a nice compact byte array going over the wire in either direction, representing a binary-serialized, compressed DataSet.

Steve from Pleasant Hill
November 21, 2008

# re: Silverlight and Data Communications

This link might have some tips using Datasets.

http://silverlight.net/forums/p/16733/55609.aspx#55609

William Plander
November 22, 2008

# re: Silverlight and Data Communications

Perry Wrote:
"Arrgh!!! I hate it when I see system with complex stored procs! All that logic should be in the middle-tier..."
."..So what do I do in this situation? Convince the client to do it properly or I walk away."

Pretty harsh there dude? Sounds to me what happens when you hire a programmer to do a databaser's job. Further argument to let programmers write the code and leave the queries in the database where they belong. The reason they invented SP's in the first place.

Ian Patrick Hughes
November 24, 2008

# re: Silverlight and Data Communications

"Further argument to let programmers write the code and leave the queries in the database where they belong."

That is always dependent upon your situation. The clamor from getting DA code out of sprocs into a business tier/layer frequently has to do with maintaining code versioning in source control and transparency when it comes to performance tuning.

DataSets are quickly becoming passé because it's so nice to work in strong typed LINQ objects these days, for even intellisense support if nothing else. However, it's no reason to throw out trustworthy tools from the past. Everything will always look like a nail if you only carry a hammer, right?

Web Development Community
November 25, 2008

# Silverlight and Data Communications - Rick Strahl's Web Log

You are voted (great) - Trackback from Web Development Community

Helen Warn
November 26, 2008

# re: Silverlight and Data Communications

I gotta say I find this so frustrating. Damien Guide (http://damieng.com/blog/2008/09/14/linq-to-sql-template-for-visual-studio-2008) has provided a nice template for generating entities that can be serialized from Linq to Sql data models, even if they contain circular references. Alas, the generated classes won't compile in Silverlight.

J'Man
December 10, 2008

# re: Silverlight and Data Communications

Rick great questions and many options here. But really none that give themselves to simplicity. Also Lot's of points of failure. I think Silverlight 2.0 is more of a "rush" to release, and we'll work "other things" in later, similar to “1.0”. Really the only conclusion I can come to at this point is the following:

Silverlight is not ready for creating real business applications, at least not yet.

Maybe it's ok to show some flashy images, interesting effects and maybe pull some basic data ("firstname", "lastname") from a service call and show it. But at the core it's missing the essential properties you speak of Rick. Where is the dynamic ability that people have used to create interesting applications? For example, we have a page we call "Experiment Query Tool" that allows one to build "on the fly" a query thru various widgets and then execute that query and return the results. We utilize logical conjunction (and nesting) to build the queries based on the fields they choose. How can I do this when I have to "know" ahead of time the contract to define for communicating with Silverlight (Can you imagine the combination). I don’t think Silverlight it is really ready for that type of application.

On a side note here:
Perry said, "data tier is just for CRUD operations" or "I will walk away", is really someone that has not really had a lot of experience with building real applications. I am always skeptical of statements like that. I have seen too many "failures" with that type attitude. For example, let's say I need to identify some products based on a condition for "mark down" and I have 6 millions records in the database. So I should pull 6 million records back in the business logic and scan thru them, instead of using a stored procedure and utilizing indexes. I try to use the tools available instead of saying “It must be this way!” But I don’t want to argue the point there, I’m about the delivering “business value” to the business versus, “You have designed it wrong”. I try to design for change because I know that is something I can count on as being constant.

Tom Van den Eynde
January 13, 2009

# re: Silverlight and Data Communications

I can imagine that supporting an old Flash application could be problematic but why didn't you turn to Adobe Flex? It's way easier to convert an application to Flex than it is to Silverlight (especially if you're not really familiar with any of the technologies).

Rick Strahl
January 13, 2009

# re: Silverlight and Data Communications

@Tom - I've used Flex a bit a while back and I'm not terribly excited about it. It works and it's easy to get started with, but for more complex logic it's not what I would call a productive environment.

With Silverlight we can at least leverage our .NET skills and tools even if the design portion of it involves a bigger learning curve than Flex. At same time though you have much more flexibility if needed.

The other issue is that the old Flash app was a custom framework, so it was actually (for the flash folks) fairly easy to work with for form layout and in fact more flexible that what flex provides even. When the decision was made to start over the MS Stack won out for consistency across client and server, and the existing skill set of devs.

Not saying that Flex is a bad choice but in our scenario it wouldn't have made much sense.

Jeff
March 26, 2009

# re: Silverlight and Data Communications

Hi Rick - What do you think about .NET RIA Services??

Jeff E.
May 07, 2009

# re: Silverlight and Data Communications

Hi Rick,

I'm facing the same problem as yours. Did do find an acceptable solution ?

Telmo
July 14, 2009

# re: Silverlight and Data Communications

Hi Rick,

Almost one year later when you first posted this, Silverlight 3 is here and ADO.NET services and WCF are still in the race with LINQ working its way in the middle (somewhere), and I have yet to find the future of data connectivity for Silverlight.

ASP and ASP.NET were pretty clear, record sets for speed, datasets for in memory processing, choices were there, pick and choose whichever one. Regardless of your choice, you knew you could work with them from the code side of things, change the queries as you went along, easily, efforteless and safely as changes to that portion of the code would not affect other data areas. So now we have WCF/ADO.NET Services to help us...

Perhaps I have been working in the wrong projects/applications for the last 20 years but I have yet to see a data to class to front end design where strict class object design will give you everything you need to work with the data. Let's name some valiant efforts: SAP, Siebel, JDE Edwards. All business object oriented applications filled with exceptions and workarounds to bypass the business logic layer.

So ultimately we revert back to XML, which can be easily expanded as needed (yet remaining structured), mechanism for data transfer. But if we do that, why exactly do we need Silverlight?

Or do we stick to web services (and again why WCF ?) and have two projects open all the time and hope that changes to the data service does not affect other areas, and ultimately end up with several web services, each with tens (if not hundreds) of methods.

Ahh.... I miss recordsets. I miss an answer here too!

Vitaly laskarzhevsky
July 22, 2009

# re: Silverlight and Data Communications

Hi Rick,
Could you look at this: http://silverlightdataset.codeplex.com/
Best regards
Vitaly

iconboy
October 06, 2009

# re: Silverlight and Data Communications

Hey Rick, the solution put together at http://www.silverlightds.com/ is more than likely what you're looking for. Let us know what you think!

Ali
March 30, 2010

# re: Silverlight and Data Communications

hey hi you guys great blog... But i am a new bee in silverlight and was wondering if it is possible to use dataset in silverlight as i have grown a comfort to it....and if not what can be done

Dave
April 12, 2010

# re: Silverlight and Data Communications

So under the new, correct, way of doing things, every change to the data schema would require code to be modified, built and deployed. Under my outdated, no longer correct, paradigm, I was able to stabilize the application with enough extensibility to introduce schema changes at run time and even allow my customers to do so without having to go back to Visual Studio. Is this really about making things better for me and my customers or just making things better for Microsoft?
 

West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2019