Read single record at a time with System.Text.Json

Question

I am trying to read a single json record into memory at a time using .net core 3.0.

This page: https://devblogs.microsoft.com/dotnet/try-the-new-system-text-json-apis/

Gives this example using a reader:

byte[] data = Encoding.UTF8.GetBytes(json);
Utf8JsonReader reader = new Utf8JsonReader(data, isFinalBlock: true, state: default);

while (reader.Read())
{
    Console.Write(reader.TokenType);

    switch (reader.TokenType)
    {
        case JsonTokenType.PropertyName:
        case JsonTokenType.String:
        {
            string text = reader.GetString();
            Console.Write(" ");
            Console.Write(text);
            break;
        }

        case JsonTokenType.Number:
        {
            int value = reader.GetInt32();
            Console.Write(" ");
            Console.Write(value);
            break;
        }

        // Other token types elided for brevity
    }

    Console.WriteLine();
}

In the example it loads the entire byte array. My main concern is memory as I am dealing with large json files, I don't want to load all of it just the current record being worked on (or at least a smaller chunk).

I am not sure how to pass a byte stream to Utf8JsonReader and read one record at a time.

What is simple way to read one record at a time with .net core 3.0?

Answer 1

The way to achieve this form of functionality would be to use JSON Lines file format with StreamReader class. JSON Lines file extension is .jsonl.

This amends the JSON string with a new line character after every JSON object. With this you can use StreamReader.ReadLine and then just deserialize the entire line.

See JSON Lines http://jsonlines.org/ for more details.