Operation is not valid due to the current state of the object (System.Text.Json)

Question

We've got an API, which simply posts incoming JSON documents to a message bus, having assigned a GUID to each. We're upgrading from .Net Core 2.2 to 3.1 and were aiming to replace NewtonSoft with the new System.Text.Json library.

We deserialise the incoming document, assign the GUID to one of the fields and then reserialise before sending to the message bus. Unfortunately, the reserialisation is failing with the exception Operation is not valid due to the current state of the object.

Here's a controller that shows the problem:-

using System;
using System.Net;
using Project.Models;
using Microsoft.AspNetCore.Mvc;
using System.IO;
using System.Text;
using System.Text.Json;

namespace Project.Controllers
{
    [Route("api/test")]
    public class TestController : Controller
    {
        private const string JSONAPIMIMETYPE = "application/vnd.api+json";

        public TestController()
        {
        }

        [HttpPost("{eventType}")]
        public async System.Threading.Tasks.Task<IActionResult> ProcessEventAsync([FromRoute] string eventType)
        {
            try
            {
                JsonApiMessage payload;

                using (StreamReader reader = new StreamReader(Request.Body, Encoding.UTF8)) {
                    string payloadString = await reader.ReadToEndAsync();

                    try {
                        payload = JsonSerializer.Deserialize<JsonApiMessage>(payloadString);
                    }
                    catch (Exception ex) {
                        return StatusCode((int)HttpStatusCode.BadRequest);
                    }
                }

                if ( ! Request.ContentType.Contains(JSONAPIMIMETYPE) )
                {
                    return StatusCode((int)HttpStatusCode.UnsupportedMediaType);
                }

                Guid messageID = Guid.NewGuid();
                payload.Data.Id = messageID.ToString();

                // we would send the message here but for this test, just reserialise it
                string reserialisedPayload = JsonSerializer.Serialize(payload);

                Request.HttpContext.Response.ContentType = JSONAPIMIMETYPE;
                return Accepted(payload);
            }
            catch (Exception ex) 
            {
                return StatusCode((int)HttpStatusCode.InternalServerError);
            }
        }
    }
}

The JsonApiMessage object is defined like this:-

using System.Text.Json;
using System.Text.Json.Serialization;

namespace Project.Models
{
    public class JsonApiMessage
    {
        [JsonPropertyName("data")]
        public JsonApiData Data { get; set; }

        [JsonPropertyName("included")]
        public JsonApiData[] Included { get; set; }
    }

    public class JsonApiData
    {
        [JsonPropertyName("type")]
        public string Type { get; set; }

        [JsonPropertyName("id")]
        public string Id { get; set; }

        [JsonPropertyName("attributes")]
        public JsonElement Attributes { get; set; }

        [JsonPropertyName("meta")]
        public JsonElement Meta { get; set; }

        [JsonPropertyName("relationships")]
        public JsonElement Relationships { get; set; }
    }
}

An example call looks like this:-

POST http://localhost:5000/api/test/event
Content-Type: application/vnd.api+json; charset=UTF-8

{
  "data": {
    "type": "test",
    "attributes": {
      "source": "postman",
      "instance": "jg",
      "level": "INFO",
      "message": "If this comes back with an ID, the API is probably working"
    }
  }
}

When I examine the contents of payload at a breakpoint in Visual Studio, it looks OK at the top level but the JsonElement bits look opaque, so I don't know if they've been parsed properly. Their structure can vary, so we only care that they are valid JSON. In the old NewtonSoft version, they were JObjects.

After the GUID has been added, it appears in the payload object when examined at a breakpoint but I'm suspicious that the problem is related to other elements in the object being read-only or something similar.

Answer 1

Your problem can be reproduced with the following more minimal example. Define the following model:

public class JsonApiMessage
{
    public JsonElement data { get; set; }
}

Then attempt to deserialize and re-serialize an empty JSON object like so:

var payload = JsonSerializer.Deserialize<JsonApiMessage>("{}");
var newJson = JsonSerializer.Serialize(payload, new JsonSerializerOptions { WriteIndented = true });

And you will get an exception (demo fiddle #1 here):

System.InvalidOperationException: Operation is not valid due to the current state of the object.
   at System.Text.Json.JsonElement.WriteTo(Utf8JsonWriter writer)
   at System.Text.Json.Serialization.Converters.JsonConverterJsonElement.Write(Utf8JsonWriter writer, JsonElement value, JsonSerializerOptions options)

The problem seems to be that JsonElement is a struct, and the default value for this struct can't be serialized. In fact, simply doing JsonSerializer.Serialize(new JsonElement()); throws the same exception (demo fiddle #2 here). (This contrasts with JObject which is a reference type whose default value is, of course, null.)

So, what are your options? You could make all your JsonElement properties be nullable, and set IgnoreNullValues = true while re-serializing:

public class JsonApiData
{
    [JsonPropertyName("type")]
    public string Type { get; set; }

    [JsonPropertyName("id")]
    public string Id { get; set; }

    [JsonPropertyName("attributes")]
    public JsonElement? Attributes { get; set; }

    [JsonPropertyName("meta")]
    public JsonElement? Meta { get; set; }

    [JsonPropertyName("relationships")]
    public JsonElement? Relationships { get; set; }
}

And then:

var reserialisedPayload  = JsonSerializer.Serialize(payload, new JsonSerializerOptions { IgnoreNullValues = true });

Demo fiddle #3 here.

Or, in .NET 5 or later, you could mark all of your JsonElement properties with [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]:

public class JsonApiData
{
    // Remainder unchanged

    [JsonPropertyName("attributes")]
    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
    public JsonElement Attributes { get; set; }

    [JsonPropertyName("meta")]
    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
    public JsonElement Meta { get; set; }

    [JsonPropertyName("relationships")]
    [JsonIgnore(Condition = JsonIgnoreCondition.WhenWritingDefault)]
    public JsonElement Relationships { get; set; }
}

Doing so will cause uninitialized elements to be skipped during serialization without needing to modify serialization options.

Demo fiddle #4 here.

Or, you could simplify your data model by binding all the JSON properties other than Id to a JsonExtensionData property like so:

public class JsonApiData
{
    [JsonPropertyName("id")]
    public string Id { get; set; }

    [JsonExtensionData]
    public Dictionary<string, JsonElement> ExtensionData { get; set; }
}

This approach avoids the need to manually set IgnoreNullValues when re-serializing, and thus ASP.NET Core will re-serialize the model correctly automatically.

Demo fiddle #5 here.

Answer 2

The exception is right - the state of the object is invalid. The Meta and Relasionships elements are non-nullable but the JSON string doesn't contain them. The deserialized object ends up with Undefined values in those properties that can't be serialized.

    [JsonPropertyName("meta")]
    public JsonElement? Meta { get; set; }

    [JsonPropertyName("relationships")]
    public JsonElement? Relationships { get; set; }

The quick fix would be to change those properties to JsonElement?. This will allow correct deserialization and serialization. By default, the missing elements will be emitted as nulls:

"meta": null,
"relationships": null

To ignore them, add the IgnoreNullValues =true option :

var newJson = JsonSerializer.Serialize(payload, new JsonSerializerOptions 
                           { WriteIndented = true,IgnoreNullValues =true });

The real solution though would be to get rid of all that code. It hampers the use of System.Text.Json. Left by itself, ASP.NET Core uses Pipelines to read the input stream without allocating, deserializes the payload and calls the method with the deserialized object as a parameter, using minimal allocations. Any return values are serialized in the same way.

The question's code though allocates a lot - it caches the input in the StreamReader, then the entire payload is cached in the payloadString and then again, as the payload object. The reverse process also uses temporary strings. This code takes at least twice as much RAM as ASP.NET Core would use.

The action code should be just :

[HttpPost("{eventType}")]
public async Task<IActionResult> ProcessEventAsync([FromRoute] string eventType,
                                                   MyApiData payload)
{
    Guid messageID = Guid.NewGuid();
    payload.Data.Id = messageID.ToString();

    return Accepted(payload);
}

Where MyApiData is a strongly-typed object. The shape of the Json example corresponds to :

public class Attributes
{
    public string source { get; set; }
    public string instance { get; set; }
    public string level { get; set; }
    public string message { get; set; }
}

public class Data
{
    public string type { get; set; }
    public Attributes attributes { get; set; }
}

public class MyApiData
{
    public Data data { get; set; }
    public Data[] included {get;set;}
}

All other checks are performed by ASP.NET Core itself - ASP.NET Core will reject any POST that doesn't have the correct MIME type. It will return a 400 if the request is badly formatted. It will return a 500 if the code throws