Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Writing JSON to a stream without buffering the string in memory

Tags:

json

c#

.net

io

I would like to write JSON to a Stream by building the document up explicitly. For example:

var stream = ...;
var writer = new JsonWriter(stream);

writer.BeginArray();
{
  writer.BeginObject();
  {
    writer.String("foo");
    writer.Number(1);
    writer.String("bar");
    writer.Number(2.3);
  }
  writer.EndObject();
}
writer.EndArray();

This would produce:

[
  {
    "foo": 1,
    "bar": 2.3
  }
]

The benefit of this approach is that nothing needs to be buffered in memory. In my situation, I'm writing quite a lot of JSON to the stream. Solutions such as this one involve creating all your objects in memory, then serialising them to a large string in memory, then finally writing this string to the stream and garbage collecting, probably from the LOH. I want to keep my memory use low, writing out elements while reading data from another file/DB/etc stream.

This kind of approach is available in C++ via the rapidjson library.

I've searched around a fair bit for this and haven't found a solution.

like image 225
Drew Noakes Avatar asked Aug 16 '13 09:08

Drew Noakes


People also ask

How to speed up JSON serialization?

Speed up serialization Because ArduinoJson writes bytes one by one, WiFiClient spends a lot of time sending small packets. To speed up your program, you need to insert a buffer between serializeJson() and WiFiClient . You can do that using the StreamUtils library.

Can JSON be streamed?

JSON streaming comprises communications protocols to delimit JSON objects built upon lower-level stream-oriented protocols (such as TCP), that ensures individual JSON objects are recognized, when the server and clients use the same one (e.g. implicitly coded in).

How do I parse a large JSON file?

There are some excellent libraries for parsing large JSON files with minimal resources. One is the popular GSON library. It gets at the same effect of parsing the file as both stream and object. It handles each record as it passes, then discards the stream, keeping memory usage low.


1 Answers

Turns out I needed to Google for a bit longer.

JSON.NET does indeed support this via its JsonWriter class.

My example would be written:

Stream stream = ...;

using (var streamWriter = new StreamWriter(stream))
using (var writer = new JsonTextWriter(streamWriter))
{
    writer.Formatting = Formatting.Indented;

    writer.WriteStartArray();
    {
        writer.WriteStartObject();
        {
            writer.WritePropertyName("foo");
            writer.WriteValue(1);
            writer.WritePropertyName("bar");
            writer.WriteValue(2.3);
        }
        writer.WriteEndObject();
    }
    writer.WriteEndArray();
}
like image 184
Drew Noakes Avatar answered Sep 19 '22 23:09

Drew Noakes