Asynchronous protobuf serialization

Question

A limitation of the protobuf-net implementation is that it calls the underlying streams synchronously. By not offering an asynchronous API e.g. BeginSerialize/EndSerialize or a TPL equivalent, we are forced to tie up a thread waiting for synchronous stream I/O.

Is there any plan to offer asynchronous methods in protobuf-net, or alternatively, any creative ways around this problem?

Answer 1

No, that isn't currently supported and would be a lot of work.

My suggestion would be: buffer data yourself using the async APIs, and then when you have the data, use something like a MemoryStream to deserialize...

In my defence, I'm not aware of any other serializer that offers an async API here. In particular, when talking about slow/async streams, that usually means "network": and you usually have the issue of "framing" to consider there; protobuf-net won't know your framing requirements...

Answer 2

I am using protobuff over the network. While the following solution doesn't guarantee it wont block, it does make life far better:

byte[] emptyByteArray = new Byte[0];
await stream.ReadAsync(emptyByteArray, 0, 0);
TaskData d = Serializer.DeserializeWithLengthPrefix<TaskData>(stream, PrefixStyle.Base128);

Because we make sure there is actual data on the stream before we start to deserialize, it will only end up blocking when the stream contains a partial message.

Edit: And we can use a similar trick for serialization:

MemoryStream mstm = new MemoryStream();
Serializer.SerializeWithLengthPrefix(mstm, data, PrefixStyle.Base128);
await stream.WriteAsync(mstm.GetBuffer(), 0, (int)mstm.Position);

As a bonus, this one does guarantee to never block.

Answer 3

You can await Task.Run that will run the synchronous code in the thread pool. It's not the most efficient solution, but it's better than blocking. You can even submit your own CancellationToken to Task.Run:

await Task.Run(() => Serializer.SerializeWithLengthPrefix(
    stream, data, PrefixStyle.Base128), cancellationToken);

Alternatively, you can use a fairly simple helper method ripped from my JuiceStream library that I have submitted as part of async feature request to protobuf-net:

await ProtobufEx.SerializeWithLengthPrefixAsync(
    stream, data, PrefixStyle.Base128, cancellationToken);
await ProtobufEx.DeserializeWithLengthPrefixAsync<MyType>(
    stream, PrefixStyle.Base128, cancellationToken);