Modifying a JSON file using System.Text.Json

Question

I know you can do this easily with Newtonsoft. As I am working with .NET Core 3.0, however, I am trying to use the new methods for interacting with JSON files —i.e., System.Text.Json—and I refuse to believe that what I am trying to do is all that difficult!

My application needs to list users that have not already been added to my database. In order to get the full list of all users, the app retrieves a JSON string from a web API. I now need to cycle through each of these users and check if they have already been added to my application before returning a new JSON list to my view so that it can display the new potential users to the end user.

As I am ultimately returning another JSON at the end of the process, I don't especially want to bother deserializing it to a model. Note that the structure of data from the API could change, but it will always have a key from which I can compare to my database records.

My code currently looks like this:

using (WebClient wc = new WebClient())
{
    var rawJsonDownload = wc.DownloadString("WEB API CALL");
    var users =  JsonSerializer.Deserialize<List<UserObject>>(rawJsonDownload);

    foreach (var user in users.ToList())
    {
        //Check if User is new
        if (CHECKS)
        {
            users.Remove(user);
        }
    }

    return Json(users); 
}

This seems like a lot of hoops to jump through in order to achieve something that would be fairly trivial with Newtonsoft.

Can anyone advise me on a better way of doing this—and, ideally, without the need for the UserObject?

Answer 1

Your problem is that you would like to retrieve, filter, and pass along some JSON without needing to define a complete data model for that JSON. With Json.NET, you could use LINQ to JSON for this purpose. Your question is, can this currently be solved as easily with System.Text.Json?

As of .NET 6, this cannot be done quite as easily with System.Text.Json because it has no support for JSONPath which is often quite convenient in such applications. There is currently an open issue Add JsonPath support to JsonDocument/JsonElement #41537 tracking this.

That being said, imagine you have the following JSON:

[
  {
    "id": 1,
    "name": "name 1",
    "address": {
      "Line1": "line 1",
      "Line2": "line 2"
    },
    // More properties omitted
  }
  //, Other array entries omitted
]

And some Predicate<long> shouldSkip filter method indicating whether an entry with a specific id should not be returned, corresponding to CHECKS in your question. What are your options?

In .NET 6 and later you could parse your JSON to a JsonNode, edit its contents, and return the modified JSON. A JsonNode represents an editable JSON Document Object Model and thus most closely corresponds to Newtonsoft's JToken hierarchy.

The following code shows an example of this:

var root = JsonNode.Parse(rawJsonDownload).AsArray(); // AsArray() throws if the root node is not an array.
for (int i = root.Count - 1; i >= 0; i--)
{
    if (shouldSkip(root[i].AsObject()["id"].GetValue<long>()))
        root.RemoveAt(i);
}

return Json(root);

Mockup fiddle #1 here

In .NET Core 3.x and later, you could parse to a JsonDocument and return some filtered set of JsonElement nodes. This works well if the filtering logic is very simple and you don't need to modify the JSON in any other way. But do note the following limitations of JsonDocument:

• JsonDocument and JsonElement are read-only. They can be used only to examine JSON values, not to modify or create JSON values.

• JsonDocument is disposable, and in fact must needs be disposed to minimize the impact of the garbage collector (GC) in high-usage scenarios, according to the docs. In order to return a JsonElement you must clone it.

The filtering scenario in the question is simple enough that the following code can be used:

using var usersDocument = JsonDocument.Parse(rawJsonDownload);
var users = usersDocument.RootElement.EnumerateArray()
    .Where(e => !shouldSkip(e.GetProperty("id").GetInt64()))
    .Select(e => e.Clone())
    .ToList();

return Json(users);

Mockup fiddle #2 here.

In any version, you could create a partial data model that deserializes only the properties you need for filtering, with the remaining JSON bound to a [JsonExtensionDataAttribute] property. This should allow you to implement the necessary filtering without needing to hardcode an entire data model.

To do this, define the following model:

public class UserObject
{
    [JsonPropertyName("id")]
    public long Id { get; set; }
    
    [System.Text.Json.Serialization.JsonExtensionDataAttribute]
    public IDictionary<string, object> ExtensionData { get; set; }
}

And deserialize and filter as follows:

var users = JsonSerializer.Deserialize<List<UserObject>>(rawJsonDownload);
users.RemoveAll(u => shouldSkip(u.Id));

return Json(users);

This approach ensures that properties relevant to filtering can be deserialized appropriately without needing to make any assumptions about the remainder of the JSON. While this isn't as quite easy as using LINQ to JSON, the total code complexity is bounded by the complexity of the filtering checks, not the complexity of the JSON. And in fact my opinion is that this approach is, in practice, a little easier to work with than the JsonDocument approach because it makes it somewhat easier to inject modifications to the JSON if required later.

Mockup fiddle #3 here.

No matter which you choose, you might consider ditching WebClient for HttpClient and using async deserialization. E.g.:

var httpClient = new HttpClient(); // Cache statically and reuse in production
var root = await httpClient.GetFromJsonAsync<JsonArray>("WEB API CALL");

Or

using var usersDocument = await JsonDocument.ParseAsync(await httpClient.GetStreamAsync("WEB API CALL"));

Or

var users = await JsonSerializer.DeserializeAsync<List<UserObject>>(await httpClient.GetStreamAsync("WEB API CALL"));

You would need to convert your API method to be async as well.