Newtonsoft Converter FromJson - unexpected token

Question

I've been trying to write a JSON deserialiser for a while now, but haven't been able to find my error. Why is Newtonsoft telling me, Unexpected token when deserializing object: StartObject, after deserialising this?

type ThisFails =
  { a : string * string
    m : Map<string, string> }

type ThisWorks =
  { y : Map<string, string>
    z : string * string }

testCase "failing test - array before object" <| fun _ ->
  let res = deserialise<ThisFails> Serialisation.converters
                                   """{"a":["xyz","zyx"],"m":{}}"""
  Assert.Equal("should be eq to res", { a = "xyz", "zyx"; m = Map.empty }, res)

testCase "passing test - array after object" <| fun _ ->
  let res = deserialise<ThisWorks> Serialisation.converters
                                   """{"y":{},"z":["xyz","zyx"]}"""
  Assert.Equal("should be eq to res", { y = Map.empty; z = "xyz", "zyx" }, res)

The subject is the TupleArrayConverter.

The trace of that converter is:

reading json [Newtonsoft.Json.FSharp.TupleArrayConverter]
  type => System.Tuple`2[System.String,System.String]

value token, pre-deserialise [Newtonsoft.Json.FSharp.TupleArrayConverter]
  path => "a[0]"
  token_type => String

value token, post-deserialise [Newtonsoft.Json.FSharp.TupleArrayConverter]
  path => "a[1]"
  token_type => String

value token, pre-deserialise [Newtonsoft.Json.FSharp.TupleArrayConverter]
  path => "a[1]"
  token_type => String

value token, post-deserialise [Newtonsoft.Json.FSharp.TupleArrayConverter]
  path => "a"
  token_type => EndArray

after EndArray token, returning [Newtonsoft.Json.FSharp.TupleArrayConverter]
  path => "m"
  token_type => PropertyName

In the converter, I'm consuming the last token, the end array, as you can see in the terminating case:

match reader.TokenType with
| JsonToken.EndArray ->
  read JsonToken.EndArray |> req |> ignore
  acc

And I'm consuming the StartArray token in the beginning...

So: why isn't this code working? (Newtonsoft.Json 6.0.8)

This is the error:

map tests/failing test - array before object: Exception: Newtonsoft.Json.JsonSerializationException: Unexpected token when deserializing object: StartObject. Path 'm', line 1, position 24.
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ResolvePropertyAndCreatorValues (Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, Newtonsoft.Json.JsonReader reader, System.Type objectType, IDictionary`2& extensionData) [0x00000] in <filename unknown>:0
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObjectUsingCreatorWithParameters (Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract contract, Newtonsoft.Json.Serialization.JsonProperty containerProperty, Newtonsoft.Json.Serialization.ObjectConstructor`1 creator, System.String id) [0x00000] in <filename unknown>:0
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateNewObject (Newtonsoft.Json.JsonReader reader, Newtonsoft.Json.Serialization.JsonObjectContract objectContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, Newtonsoft.Json.Serialization.JsonProperty containerProperty, System.String id, System.Boolean& createdFromNonDefaultCreator) [0x00000] in <filename unknown>:0
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateObject (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x00000] in <filename unknown>:0
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.CreateValueInternal (Newtonsoft.Json.JsonReader reader, System.Type objectType, Newtonsoft.Json.Serialization.JsonContract contract, Newtonsoft.Json.Serialization.JsonProperty member, Newtonsoft.Json.Serialization.JsonContainerContract containerContract, Newtonsoft.Json.Serialization.JsonProperty containerMember, System.Object existingValue) [0x00000] in <filename unknown>:0
  at Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize (Newtonsoft.Json.JsonReader reader, System.Type objectType, Boolean checkAdditionalContent) [0x00000] in <filename unknown>:0  (00:00:00.1500996)
1 tests run: 0 passed, 0 ignored, 0 failed, 1 errored (00:00:00.2482623)

Answer 1

Been debugging your code together with JSON.Net.

It turns out that after you consumed the EndArray token in your failing case the reader then points to a PropertyName token, that is all good.

Then after your converter has completed execution JSON.Net executes this.

} while (!exit && reader.Read());

Read() then moves the reader to the next token which in your failing case is StartObject causing the deserializer to fail.

So, I am not an expert at JSON.Net but thinking about providing a provider for a string value in JSON.Net I probably wouldn't advance the reader after conversion meaning the reader still points to the string value. Along the same line of thinking it makes sense when consuming an array to leave the reader at last token of the array value ie the EndArray token.

So my suggestion is merely this:

match reader.TokenType with
| JsonToken.EndArray ->
//    read JsonToken.EndArray |> req |> ignore

  Logger.debug logger <| fun _ ->
    LogLine.sprintf
      [ "path", reader.Path |> box
        "token_type", reader.TokenType |> box ]
      "after EndArray token, returning"

  acc

This makes my test program:

[<EntryPoint>]
let main argv = 
    let works = deserialize<ThisWorks> """{"y":{},"z":["xyz","zyx"]}"""
    printfn "%A" works

    let fails = deserialize<ThisFails> """{"a":["xyz","zyx"],"m":{}}"""
    printfn "%A" fails

    0

Print

{y = map [];
 z = ("xyz", "zyx");}
{a = ("xyz", "zyx");
 m = map [];}

Hopefully this helps you resolve this error (you might already have done so)