Serialize Newtonsoft JSON to byte array

Question

What I am aiming to do is send JSON containing a header object and a compressed data in a field that is byte array.

[JsonObject(MemberSerialization.OptOut)]
public class Message
{
    public Message()
    {
        Header = new Header();
    }

    public Header Header { get; set; }


    public byte[] Data { get; set; }
}

Byte array is gzip compressed JSON object, but this is not that relevant. Issue I am having is that if I serialize the JSON it gets converted into string and then back to bytes. Issue is, the message size increases quite a bit , since serializing the byte array converts it to string representation.

I am constrained by maximum message size and I have spiting of compressed data in place, but when I get to send the JSON containing compressed data in Byte array and Uncompressed header, serializing JSON object puts me way over the message size limit.

Is there any reliable way of converting JSON object to byte array straight away.

var stringMessage = JsonConvert.SerializeObject(message,Formatting.None);
var bytes = Encoding.UTF8.GetBytes(stringMessage);

var stringMessage2 = JsonConvert.SerializeObject(message.TransportHeader, Formatting.None);
var bytes2 = Encoding.UTF8.GetBytes(stringMessage2);

Message eventMessage = new Message(bytes);
var bytes3= Encoding.UTF8.GetBytes(JsonConvert.SerializeObject(message.Transportdata));

Compressed data size =243905

Full JSON in Bytes after serialization = 325313

Just header in bytes size =90

Just Compressed data serialized and converted back to bytes = 325210, (size increases when data gets serialized by JsonConvert.SerializeObject and string representation is produced)

It clearly goes up quite a bit and its caused by byte array.

Answer 1

I found a way to do what I wanted, its not precisely JSON, buts is BSON or also known as Binary JSON. Well since finding the solution was a pure luck, I am not sure how well known is BSON.

Anyway Newtonsoft supports it via Newtonsoft.Json.Bson nuget package at https://www.nuget.org/packages/Newtonsoft.Json.Bson/1.0.1

Some code for serialization/deserialization

foreach (var message in transportMessageList)
{
    MemoryStream ms = new MemoryStream();
    using (BsonDataWriter writer = new BsonDataWriter(ms))
    {
        JsonSerializer serializer = new JsonSerializer();
        serializer.Serialize(writer, message);
    }

    var bsonByteArray = ms.ToArray();
    Assert.True(bsonByteArray.Length!=0);
    bsonList.Add(bsonByteArray);
}

var deserializdTransmortMessageList = new List<TransportMessage>();
foreach (var byteArray in bsonList)
{
    TransportMessage message;
    MemoryStream ms = new MemoryStream(byteArray);
    using (BsonDataReader reader = new BsonDataReader(ms))
    {
        JsonSerializer serializer = new JsonSerializer();
        message = serializer.Deserialize<TransportMessage>(reader);
    }
    Assert.True(message.Transportdata.Length!=0);
    deserializdTransmortMessageList.Add(message);
}

You can use same classes/objects you use for JSON, serializing compressed array of bytes no longer cause an increase in size.

Please note that BSON documentation at Newtonsoft website is out dated and lists only deprecated api calls at this moment. My code uses proper non deprecated API calls.

Answer 2

JSON is a character based format so there is necessarily character data involved. I suspect you used UTF16 encoding which makes each char into two bytes. If you use UTF8 you will not experience any meaningful size overhead.