Rick Strahl's Weblog  

Wind, waves, code and everything in between...
.NET • C# • Markdown • WPF • All Things Web
Contact   •   Articles   •   Products   •   Support   •   Advertise
Sponsored by:
Markdown Monster - The Markdown Editor for Windows

JSON and Date Embedding


:P
On this page:

I'm a bit mystified by various JSON implementations out there that don't seem to be dealing with dates correctly. I've been using Douglas Crockford's JSON implementation from JSON.org and originally I had to fix up the date processing so that it would work on both ends for serializing and deserializing. I noticed today that Douglas has actually updated the code recently and unlike the old code there's at least some thought given to date formatting but it still doesn't 'just' work.

I checked a couple of other client side JavaScript libraries including jQuery .ToJSON which also doesn't work - it just creates an empty object and fails to completely recognize dates. I mean are these people just not testing their converters with dates or am I missing something horribly obvious? <s>

Douglas Crockford's latest code does something like this for date encoding:

Date.prototype.toJSON = function () {

// Eventually, this method will be based on the date.toISOString method.

    return this.getUTCFullYear()   + '-' +
         f(this.getUTCMonth() + 1) + '-' +
         f(this.getUTCDate())      + 'T' +
         f(this.getUTCHours())     + ':' +
         f(this.getUTCMinutes())   + ':' +
         f(this.getUTCSeconds())   + 'Z';
};

That works fine on the server end with the date format, but unfortunately on the clientside doing something like this fails:

var temp = { Name: "Rick", Entered: new Date() };

var txt = JSON.stringify( temp );
alert( txt );
        
var temp2 = JSON.parse(txt);
alert( temp2.Entered);  // string display instead of Date

The problem is that the date essentially generates as a string into the JSON output, which doesn't really do anything when the output is is

Now in my own library (which use the old version of Douglas Crockford's JSON code) I have a short snippet of code added that generates a new date object like this: (part of the wwHoverPanel libray)

case 'object':
  if (x) {
    // RAS: Added Date Parsing
    if (x.toUTCString)
      return e('new Date(' +     x.getUTCFullYear() + ',' + x.getUTCMonth() + ',' + x.getUTCDate() + ',' + x.getUTCHours() + ',' + x.getUTCMinutes() + ',' + x.getUTCSeconds() + ',' + x.getUTCMilliseconds() + ')' );

    ...

This generates dates like this:

new Date(2007,11,24,23,59,59,200)

right into the JSON and this code round trips just fine in JavaScript, so I can serialize() and parse() and get the same date value back as part of object parsing. When the data is sent to the server the code above can handle it.

But this is an implementation specific way of doing this I suppose, but I couldn't find anything that describes the official way to describe a date in JSON so it works for roundtripping. I noticed that MS Ajax is using Javascript Date.GetTime() style formats specify its date formatting which is based on the JavaScript GetTime() function which can be used to create a date.

To create a JavaScript compatible millisecond count since 1970:

DateTime baseDate = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);

 

// *** Convert to JavaScript specific milliseconds

TimeSpan tspan = DateTime.UtcNow.Subtract(new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc));

double ticks = tspan.TotalMilliseconds;

string jsonDate = "new Date(" + ticks.ToString() + ")";

 

 

// *** Convert back to a date

DateTime now = baseDate.AddMilliseconds(ticks).ToLocalTime();

Applied that to my client and server code and it round trips nicely and it's also a little more compact  - it'll save a few bytes in addition to requiring less code to parse.

So there a few date format that need to be considered. There's the comma syntax I originally used. There's GetTime() syntax that several other tools use and there's the string encoded values that might be used by 'rogue' AJAX client libraries. At this point I have the following date parsing code on the server:

private DateTime baseDate = new DateTime(1970, 1, 1, 0, 0, 0, DateTimeKind.Utc);

 

... parsing code

 

else if (ValueType == typeof(DateTime))

{

    if (Value == null)

        return DateTime.MinValue;

 

    try

    {                   

        // *** Parse out what's between the date parens

        Value = wwUtils.ExtractString(Value, "new Date(", ")");

 

        // new Date() should never happen - error

        if (string.IsNullOrEmpty(Value))

            return DateTime.MinValue;

 

        // *** 1999,12,1,12,00,59

        if (Value.Contains(","))

        {

            string[] DateParts = Value.Split(',');

 

            return new DateTime(int.Parse(DateParts[0]),

                                int.Parse(DateParts[1]) + 1,

                                int.Parse(DateParts[2]),

                                int.Parse(DateParts[3]),

                                int.Parse(DateParts[4]),

                                int.Parse(DateParts[5]),

                                DateTimeKind.Utc).ToLocalTime();

        }

        // *** JavaScript .GetTime() style date based on milliseconds since 1970

        if (!Value.Contains("-") && !Value.Contains("/"))

        {                       

            double ticks = 0;

            if (double.TryParse(Value,out ticks))

                return this.BaseDate.AddMilliseconds(ticks).ToLocalTime();

 

            return DateTime.MinValue;

        }

 

        // *** Try to parse with UTC round trip format or invariant auto-detection                   

        Value = Value.Replace("\"", "");

        return DateTime.Parse(Value, CultureInfo.InvariantCulture.DateTimeFormat, DateTimeStyles.RoundtripKind).ToLocalTime();                       

    }

    catch

    {

        return DateTime.MinValue;

    }

}

 

... serialization code

else if (val is DateTime)

{

    double milliseconds = 0;

 

    DateTime date = (DateTime)val;

    if (date.CompareTo(this.BaseDate) < 0)

        milliseconds = 0D;

    else

    {

        // *** Convert to JavaScript specific milliseconds

        TimeSpan tspan = ((DateTime)val).Subtract(this.BaseDate);

        milliseconds = tspan.TotalMilliseconds;

    }

 

    sb.Append("new Date(");

    sb.Append(milliseconds);

    sb.Append(")");

}

This seems to work fairly well with various different date formats for two way formatting. This code is part of JsonSerializer.cs  in the wwHoverPanel download in case you are interested in the rest of the serialization/deserialization code.

Posted in AJAX  ASP.NET  jQuery  

The Voices of Reason


 

David Betz
December 25, 2007

# re: JSON and Date Embedding


Bertrand Le Roy
December 26, 2007

# re: JSON and Date Embedding

This is simply because there is no date literal in JavaScript. Incredible omission but there it is. As JSON is supposed to be a language-neutral, pure data format, using "new Date(...)" is plain wrong. That's why in ASP.NET Ajax, we're now using a special string format that uses a small trick with some encoding of characters that is supported by the JSON specs while not strictly necessary. Putting implementation details aside, the net result is that our date format is accepted by all JSON parsers even if they don't know about your date encoding (any "new Date(...)" syntax will be chocked on by correctly implemented parsers) which will just see a string, but a parser such as ours that knows about the convention is going to be able to unambiguously interpret the date contents.

Rick Strahl
December 26, 2007

# re: JSON and Date Embedding

Bertrand - This is what you use in ASP.NET Ajax:

{"d":"\/Date(1198737665392)\/"}

where the value is the ticks. Now this works just fine if you control over both ends but if you want to use a generic JSON client (say jQuery or Prototype) they're not going to know what to do with this. In that case the alternative is to read the value as a string and parse it?

Ok, I'm going to add the parsing semantics at least <s>...

Thinking out loud here for a second. I'm not exactly sure why new Date() (or any other 'code' embedded in the JSON script) would be a problem necessarily. I mean you are already communicating with the server which can send any number of scripts into your page via script tags and loaded client script. Where's the additional security issue in this scenario? If the server wants to inject code into the client it can do so any number of much easier ways...

Bertrand Le Roy
December 27, 2007

# re: JSON and Date Embedding

It's not a question of security. It's a question of adhering to the JSON specs. Chances are, if you're using a JavaScript parser, it will use eval and new Date will just work. Well, almost, as most parsers now validate the payload using a regular expression before they eval and new Date may fail at this point. But if you're using a non-JavaScript parser, it will fail unless the implementer knew that he had to special case that. But if you adhere strictly to the specs, it's a no.
So the idea with the format we're using for dates is that it's strictly valid and standard JSON, and if you're working with that, you're fine, there can be no parser error, your dates just are interpreted as strings. If you know about this convention (and to be clear, there has to be some form of convention for dates as they are not part of the spec, there are just conventions that are more aggressive than others), you can interpret this as a date safely because of an oddity in the JSON specs which is that you may encode (quite unnecessarily) '/' as '\/'. Both forms are strictly equivalent in both JavaScript and JSON but as you parse the JSON string (or preprocess it before eval), you can transform those \/Date(...)\/ quite safely into dates as nobody actually escapes slashes and the general form is very unlikely because of that. The convention also alows for the encoding of actual strings such as "\\/Date(1198737665392)\\/" which will safely remain strings and not be interpreted as dates.
I think this is quite neat actually.

Rick Strahl
December 28, 2007

# re: JSON and Date Embedding

Yes the issue is that most inbound parsers will use regular expressions on the inbound list to strip out anything they deem invalid and code and new tends to be one of those things.

Thanks for the clarification of the format used in ASP.NET AJAX. Just for consistency I switched my code to encode that was as well. I played around with a few different things - including using toISODate() style formatting - but I do agree that using the format you chose is easiest get back into a real date as it's efficient and easily parsable. InternallyI'd just been running the eval directly but using this I ended up using JSON.parse() (Crockfords old version) and injecting some additional parse checks for the date encodings after the initial regEx validation code.

Originally I had used new Date() with ticks. For a non-JavaScript parser that's pretty easy to parse as well - certainly even easier than any special markup. At this point my parser understands: new Date(js milliseconds), ASP.NET Ajax format (default) and ISO date format. That ought to cover a good range of cases I suppose. <s> I suspect thought it's isn't going to matter much what 'outside' support there is. People are likely to use one solution on front and back for the JSON parsing as a matched set I hope.

It's a bear to deal with different frameworks. I'm trying to support my own clients, MS AJAX clients and generic AJAX clients and it actually works fairly well at this point although the outbound MS AJAX data is a pain because of the special formatting and type embedding.

Rick Strahl
December 28, 2007

# re: JSON and Date Embedding

Another slightly interesting twist: WCF's new JSON features serialize dates like this:

{"Entered":"\/Date(1198874600025-1000)\/","Name":"Rick"}

which seems really odd. The -1000 in this case is a time zone adjustment. It works correctly and is compatible with the MS Ajax format already in use (if eventually evaled that is). Apparently the timezone value is deliberately adjusted with the timezone offset (which is phoney since 1000 milliseconds is certainly not a valid offset for anything useful) . I suspect the reasoning behind this is for a parser to be able to somehow peel out a client timezone from the date. Not sure when you'd need that <shrug>

Rick Strahl
January 07, 2008

# re: JSON and Date Embedding

The following code can be used to parse JSON that includes Microsoft's date formatting from JavaScript (you can find the full JSON class in the wwHoverPanel download):

...
    // *** RAS Update: RegEx handler for dates ISO and MS AJAX style
    regExDate: function(str,p1, p2,offset,s) 
    {
        str = str.substring(1).replace('"','');
        var date = str;
        
        if (str.substring(0,7) == "\\\/Date(") {  // MS Ajax date: /Date(19834141)/       
            str = str.match(/Date\((.*?)\)/)[1];                        
            date = "new Date(" +  parseInt(str) + ")";
        }
        else { // ISO Date 2007-12-31T23:59:59Z                                     
            var matches = str.split( /[-,:,T,Z]/);        
            matches[1] = (parseInt(matches[1],0)-1).toString();                     
            date = "new Date(Date.UTC(" + matches.join(",") + "))";         
       }                  
        return date;
    },
    parse: function(text) {
        if (!(!(/[^,:{}\[\]0-9.\-+Eaeflnr-u \n\r\t]/.test(
             text.replace(/"(\\.|[^"\\])*"/g, '')))  ))
             throw new Error("Invalid characters in JSON parse string.");                 

        // *** RAS Update:  Fix up Dates: ISO and MS AJAX format support
        var regEx = /(\"\d{4}-\d{2}-\d{2}T\d{2}:\d{2}.*?\")|(\"\\\/Date\(.*?\)\\\/")/g;
        text = text.replace(regEx,this.regExDate);      

        return eval('(' + text + ')');    
    },
    parseSafe: function(text) {
        try {return this.parse(text);} catch(e) {return null;}
    }

Jack Frusciante
April 20, 2008

# re: JSON and Date Embedding

I'm trying to call an ASP.NET JSON Webservice using jQuery instead of ASP.NET AJAX Client Library. I'm trying to serialize a date using json2.js from www.json.org.

    var dat="\\/Date(1208644401687)\\/";
    
    var myData={title:"hello",data:dat};
    
    var myDataSerialized = JSON.stringify(myData);


dat ha a value of @"\/Date(1208644401687)\/" but when serialized it becomes @"\\/Date(1208644401687)\\/". I don't understand wht json serializer adds 2 backslash. It seems there's no way to serialize a date in the correct format. Any suggestion?

Rick Strahl
April 20, 2008

# re: JSON and Date Embedding

You need to double encode the value (ie. 4 slashes in C#/JavaScript string expressions)...

Jack Frusciante
April 20, 2008

# re: JSON and Date Embedding

What do you mean?
var dat="\\/Date(1208644401687)\\/";


Isn't right? Should i add some slashes?

I need dat to be \/Date(1208644401687)\/ instead of \\/Date(1208644401687)\\/ because when i try to send the XHR to the server i get an error, because the server can't deserialize this value to a DateTime type.

this is the call code:
    var dat="\\/Date(1208644401687)\\/";
    
    var myData={title:"hello",date:dat};
    
    var myDataSerialized = JSON.stringify(myData);

  $.ajax({
     type: "POST",
     url: "../myService.asmx/SaveData",
     data: "{\"mydata\":"+ myDataSerialized +"}",      
     contentType: "application/json; charset=utf-8",
     dataType: "json",    
     success: onSuccessSave,
     error: onErrorSave
   });


and this is webservice's firm

      public void SaveData(string title, DateTime data)


Thank you for your help :)

Rick Strahl
April 21, 2008

# re: JSON and Date Embedding

Ok sorry, I had to look at this again.

JSON2.js is actually doing double encoding on the date format so it's not right. I'll be posting an updated hacked JSON2.js file that works properly in another post shortly.

What you basically have to do is remove the Data.prototype.toJSON() function and instead do the conversion down in where object parsing occurs...

Tony Steele
September 10, 2008

# re: JSON and Date Embedding

In your code I have had to comment out the following test:

if (!(!(/[^,:{}\[\]0-9.\-+Eaeflnr-u \n\r\t]/.test(
text.replace(/"(\\.|[^"\\])*"/g, ''))) ))
throw new Error("Invalid characters in JSON parse string.");

as in Safari and Google Chrome this throws an error when html is returned in the result of your ajax method call. This is repeatable for certain html returned, not all html causes this error. I have not been able to determine what is causing it to fail. The same html does not throw the error when running in IE7 or Firfox.

Rick Strahl
July 25, 2009

# re: JSON and Date Embedding

@Tony - removing this code exposes you to some potential security issues. If you can provide some HTML returned in the JSON result i can take a look and see what's causing the failure...

Deepak Chawla
September 20, 2009

# re: JSON and Date Embedding

Hi Rick,
I have been using your serialisation json2.js script. But since the upgrading to firefox 3.52 I had started getting error in serilaisation date.
This was due to the following

1) This was due to change of date format from ISO Date 2007-12-31T23:59:59Z to ISO Date 2007-12-31T23:59:59.123Z with milliseconds.

2) Not only that also getting an empty last list in array hence getting new Date(2009, 01, 02, 23, 23, 23.45, ). Note the last blank. Hence erroring of getting date when using eval.

3) Finally I have also changed the month -1. parseInt(month, 0) was always giving a zero.

Following script would fixes all the issues. Will work in case of date is with or without milliseconds.

Regards,
Deepak Chawla

// *** RAS Update: RegEx handler for dates ISO and MS AJAX style
regExDate: function(str, p1, p2, offset, s) {
str = str.substring(1).replace('"', '');
var date = str;

// MS Ajax date: /Date(19834141)/
if (/\/Date(.*)\//.test(str)) {
str = str.match(/Date\((.*?)\)/)[1];
date = "new Date(" + parseInt(str) + ")";
}
/*else { // ISO Date 2007-12-31T23:59:59Z
var matches = str.split(/[-,:,T,Z]/);
matches[1] = (parseInt(matches[1], 0) - 1).toString();
date = "new Date(Date.UTC(" + matches.join(",") + "))";
}*/
// *** Deepak Chawla update
else { // ISO Date 2007-12-31T23:59:59.123Z
var matches = str.split(/[-,:,.,T,Z]/);
while (matches[matches.length - 1] == "") // Remove the last object if = ""
matches = matches.splice(0, matches.length - 1);
matches[1] = matches[1] - 1; // Removed parseInt as it was giving a value of 0
date = "new Date(Date.UTC(" + matches.join(",") + "))";
}
return date;
},

Rick Strahl
September 20, 2009

# re: JSON and Date Embedding

Take a look at this updated post which deals with dates properly:

http://www.west-wind.com/Weblog/posts/896411.aspx

There have been several follow up posts to this but the above addresses native browser ISO dates and various combination of Microsoft date formats.

Peter Grand
October 23, 2009

# re: JSON and Date Embedding

Rick -- Related to Tony Steele's comment that some html was causing the "invalid characters in json parse string" error message to be thrown in Safari only with certain html only...

I also experienced this and testing showed that the issue pertained to the size of the ajax response content. It does not show up as an issue on Safari 4 on the Mac, but it can be reproduced on Safari 3.1 on the Mac and Safari 3.2.2 on Windows. The size limit may vary between these browsers, but the same html (a table) with rows repeated a varying number of times demonstrated the issue for me.

I bracketed the "throw" as follows:
if (!jQuery.browser.safari)
{ throw new Error("Invalid characters in JSON parse string."); }
which resolved the issue but may have opened up the security concern you expressed to Tony on Safari/Chrome only.

I couldn't use code like:
jQuery.browser.safari && parseInt(jQuery.browser.version) < 417
because Safari 3.2.2 return 525 for the version. I imagine I could further parse useragent to limit bracketing out the throwing the error... but I can't yet be sure of all the Safari versions that might exhibit this issue.

For some reason
if (!noCheck && !(!(/[^,:{}\[\]0-9.\-+Eaeflnr-u \n\r\t]/.test(
text.replace(/"(\\.|[^"\\])*"/g, '')))))
returns true for those platforms for lengthy html responses.

I ddn't squelch any of the rest of the function -- just the error thrown. So what's the security risk?

Rick Strahl
October 23, 2009

# re: JSON and Date Embedding

Peter - you definitely want the RegEx expression on the JSON returned as that tries to ensure there's no executable code inside of the RegEx. My guess is that the JSON is either truncated or includes something like new Date() inside of it.

Sky
February 18, 2010

# re: JSON and Date Embedding

Hey Rick, I have been working on a service proxy that works with the /js /jsdebug client script and was struggling with the UTC issue.

I pulled down your tweaked json2 and just couldn't seem to get results. I must be borking it somewhere.

Ultimately, the cleanest solution I have found was to just munge up the date.getTime and add -0000.

// here is how we force wcf to parse as UTC and give correct local time serverside
var date = '\/Date(' + new Date().getTime() + '-0000)\/';


the .JsonReaderDelegator.ParseJsonDate method is pretty simplistic and searches for '-' or '+' and parses UTC if found.

Since javascript is always going to output UTC .getTime() it seems to me that this will give expected results server side in all cases.

Rick Strahl
February 19, 2010

# re: JSON and Date Embedding

Sky - I think I addressed this issue some time ago in the West Wind Web Toolkit. Take a look at the date code I propose here for the native JSON parsers and JSON2:

http://www.west-wind.com/weblog/posts/729630.aspx

I haven't had any issues with date conversions with this code in the West Wind Web Toolkit and a customized JSON2 implementation that I use there when native JSON isn't available. The latest version of that code is always available here (at the bottom of the library file):

http://www.west-wind.com:8080/svn/WestwindWebToolkit/trunk/Westwind.Web/Resources/ww.jquery.js

West Wind  © Rick Strahl, West Wind Technologies, 2005 - 2024