Unexpected token when deserializing object in JsonConvert.DeserializeObject

Question

I have the following test code:

[TestClass]
public class TestJsonDeserialize
{
    public class MyClass
    {
        [JsonProperty("myint")]
        public int MyInt { get; set; }
        [JsonProperty("Mybool")]
        public bool Mybool { get; set; }
    }

    [TestMethod]
    public void Test1()
    {
        var errors = new List<string>();
        var json1 = "{\"myint\":1554860000,\"Mybool\":false}";
        var json2 = "{\"myint\":3554860000,\"Mybool\":false}";
        var i = JsonConvert.DeserializeObject<MyClass>(json2, new JsonSerializerSettings
        {
            Error = delegate (object sender, Newtonsoft.Json.Serialization.ErrorEventArgs args)
            {
                Debug.WriteLine(args.ErrorContext.Error.Message);
                errors.Add(args.ErrorContext.Error.Message);
                args.ErrorContext.Handled = true;
            }
        });
        Assert.IsTrue(errors.Count <= 1);
    }
}

The call to JsonConvert.DeserializeObject produces 2 errors. One of them is expected, but the other not. The errors are:

• JSON integer 3554860000 is too large or small for an Int32. Path 'myint', line 1, position 19.
• Unexpected token when deserializing object: Boolean. Path 'Mybool', line 1, position 34.

Why is there a 2nd error although the 1st error is marked as handled. I already updated from Newtonsoft.Json 8.0.2 to 9.0.1 but it remains. When passing the first string (json1 instead of json2), then no errors at all occur.

Answer 1

Update

Reported as Issue 1194: JsonTextReader.ParseNumber leads to error after ThrowReaderError and closed by Newtonsoft as not reproducible in then-current build which subsequently was released as Json.NET 10.0.1.

Original Answer

This may be a bug in JsonTextReader.

In JsonTextReader.ParseNumber(ReadType readType, char firstChar, int initialPosition) there is the following logic, somewhat simplified:

else if (readType == ReadType.ReadAsInt32)
{

// Snip

        int value;
        ParseResult parseResult = ConvertUtils.Int32TryParse(_stringReference.Chars, _stringReference.StartIndex, _stringReference.Length, out value);
        if (parseResult == ParseResult.Success)
        {
            numberValue = value;
        }
        else if (parseResult == ParseResult.Overflow)
        {
            throw ThrowReaderError("JSON integer {0} is too large or small for an Int32.".FormatWith(CultureInfo.InvariantCulture, _stringReference.ToString()));
        }
        else
        {
            throw ThrowReaderError("Input string '{0}' is not a valid integer.".FormatWith(CultureInfo.InvariantCulture, _stringReference.ToString()));
        }
    }

    numberType = JsonToken.Integer;
}

// Snip
// Finally, after successfully parsing the number

ClearRecentString();

// index has already been updated
SetToken(numberType, numberValue, false);

At the point the exception is thrown by ThrowReadError(), the stream position has been advanced past the too-large integer. However, the value for JsonReader.TokenType has not been updated and still returns JsonToken.PropertyName for the last token that was successfully parsed, namely the "myint" name. Later, after the exception is swallowed and ignored, the inconsistency between the stream position and current token value causes the "Mybool" property name to be skipped, leading to the second error.

If, in the debugger, when the exception is thrown I manually call

SetToken(JsonToken.Undefined);
ClearRecentString();

Then the remainder of the file can be successfully parsed. (I'm not sure JsonToken.Undefined is the right choice here.)

You might want to report an issue to Newtonsoft.

Since the JsonReader is not passed in to the error handler, the only workaround I could find is to subclass JsonTextReader as follows:

public class FixedJsonTextReader : JsonTextReader
{
    public FixedJsonTextReader(TextReader reader) : base(reader) { }

    public override int? ReadAsInt32()
    {
        try
        {
            return base.ReadAsInt32();
        }
        catch (JsonReaderException)
        {
            if (TokenType == JsonToken.PropertyName)
                SetToken(JsonToken.None);
            throw;
        }
    }
}

And then do:

var errors = new List<string>();
var json2 = "{\"myint\":3554860000,\"Mybool\":false}";

using (var reader = new FixedJsonTextReader(new StringReader(json2)))
{
    var settings = new JsonSerializerSettings
    {
        Error = delegate(object sender, Newtonsoft.Json.Serialization.ErrorEventArgs args)
        {
            Debug.WriteLine(args.ErrorContext.Error.Message);
            errors.Add(args.ErrorContext.Error.Message);
            args.ErrorContext.Handled = true;
        }
    };
    var i = JsonSerializer.CreateDefault(settings).Deserialize<MyClass>(reader);
}
Assert.IsTrue(errors.Count <= 1); // Passes