Deserializing base64 encoded byte arrays return different results in 2 JSON.net library versions

Question

  JObject j = JObject.Parse("{'responseArray':'AAAAAAAAAAAAAAAAAAAAAAAAAAABAAAA'}");

  byte[] r = j["responseArray"].ToObject<byte[]>(JsonSerializer.Create(new JsonSerializerSettings()));

Expected Output (works fine with JSON.net v4.5.6.14930)

r =  {byte[24]}
        [0]: 0
        [1]: 0
        [2]: 0
        [3]: 0
        [4]: 0
        [5]: 0
        [6]: 0
        [7]: 0
        [8]: 0
        [9]: 0
        [10]: 0
        [11]: 0
        [12]: 0
        [13]: 0
        [14]: 0
        [15]: 0
        [16]: 0
        [17]: 0
        [18]: 0
        [19]: 0
        [20]: 1
        [21]: 0
        [22]: 0
        [23]: 0

Deserializing the above code in JSON.net v6.0.8.18111 gives errorneous result

Incorrect Output (JSON.net v6.0.8.18111)

r= {byte[16]}
    [0]: 170
    [1]: 170
    [2]: 170
    [3]: 170
    [4]: 170
    [5]: 170
    [6]: 170
    [7]: 170
    [8]: 170
    [9]: 170
    [10]: 170
    [11]: 170
    [12]: 170
    [13]: 171
    [14]: 170
    [15]: 170

Am I doing something wrong here?

Note- The snippet in the question is part of a complex piece of code, where data comes in as JSON parameter to a WebMethod, The responsearray is a part of a complex object sent into the webmethod, that is why we use the JSON.net library for decoding.

Answer 1

Update

This is fixed in Json.NET 7.0 Release 1.

Original Answer

This is a Json.NET issue. The problem is that your string AAAAAAAAAAAAAAAAAAAAAAAAAAABAAAA just happens to be successfully parsable as a GUID, in JsonReader.ReadAsBytesInternal():

            if (s.Length == 0)
            {
                data = new byte[0];
            }
            else if (ConvertUtils.TryConvertGuid(s, out g))
            {
                data = g.ToByteArray();
            }
            else
            {
                data = Convert.FromBase64String(s);
            }

At this traceback:

Newtonsoft.Json.JsonReader.ReadAsBytesInternal() Line 517   C#
Newtonsoft.Json.Linq.JTokenReader.ReadAsBytes() Line 74 + 0x9 bytes C#
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.ReadForType(Newtonsoft.Json.JsonReader reader = {Newtonsoft.Json.Linq.JTokenReader}, Newtonsoft.Json.Serialization.JsonContract contract = {Newtonsoft.Json.Serialization.JsonPrimitiveContract}, bool hasConverter = false) Line 1853 + 0x8 bytes   C#
Newtonsoft.Json.Serialization.JsonSerializerInternalReader.Deserialize(Newtonsoft.Json.JsonReader reader = {Newtonsoft.Json.Linq.JTokenReader}, System.Type objectType = {Name = "Byte[]" FullName = "System.Byte[]"}, bool checkAdditionalContent = false) Line 144 + 0x2f bytes   C#
Newtonsoft.Json.JsonSerializer.DeserializeInternal(Newtonsoft.Json.JsonReader reader = {Newtonsoft.Json.Linq.JTokenReader}, System.Type objectType = {Name = "Byte[]" FullName = "System.Byte[]"}) Line 710 + 0x52 bytes    C#
Newtonsoft.Json.JsonSerializer.Deserialize(Newtonsoft.Json.JsonReader reader = {Newtonsoft.Json.Linq.JTokenReader}, System.Type objectType = {Name = "Byte[]" FullName = "System.Byte[]"}) Line 689 + 0x11 bytes    C#
Newtonsoft.Json.Linq.JToken.ToObject(System.Type objectType = {Name = "Byte[]" FullName = "System.Byte[]"}, Newtonsoft.Json.JsonSerializer jsonSerializer = {Newtonsoft.Json.JsonSerializer}) Line 1837 + 0x11 bytes    C#
Newtonsoft.Json.Linq.JToken.ToObject(System.Type objectType = {Name = "Byte[]" FullName = "System.Byte[]"}) Line 1811 + 0x3c bytes  C#
Newtonsoft.Json.Linq.JToken.ToObject<byte[]>() Line 1698 + 0x39 bytes   C#

As you can see, this short-circuits the base 64 decoding. Ugly bug, and rather bad luck on your part to get that exact string.

To block Json.NET's inappropriate GUID recognition you can create your own global JsonConverter for byte arrays. When you do, Json.NET will pass you the raw string without the "helpful" GUID pattern match:

public class ByteConverter : JsonConverter
{
    public override bool CanConvert(Type objectType)
    {
        return objectType == typeof(byte[]);
    }

    public override bool CanWrite { get { return false; } } // Use the default implementation for serialization, which is not broken.

    public override object ReadJson(JsonReader reader, Type objectType, object existingValue, JsonSerializer serializer)
    {
        var value = (string)JToken.Load(reader);
        if (value == null)
            return null;
        if (value.Length == 0)
            return new byte[0];
        return Convert.FromBase64String(value);
    }

    public override void WriteJson(JsonWriter writer, object value, JsonSerializer serializer)
    {
        throw new NotImplementedException();
    }
}

And then, to set it globally:

        JsonConvert.DefaultSettings = () => new JsonSerializerSettings
        {
            Converters = new List<JsonConverter> { new ByteConverter() }
        };

This workaround restores the correct deserialization for byte arrays.