Reading Protobuf objects using boost::asio::read_async

Question

I am writing an application using Boost asio in which the client and the server exchange messages that are serialized using google proto-buffers. I do not know what is the size of the serialized message being sent on over the network. It seems that the proto-buf objects do not have any delimiter.

Here are the contents of the .proto file.

 package tutorial;

message Person {
        required string name = 1;
        required int32 id = 2;
        optional string email = 3;
}

Here is how I am writing from the server

        tutorial::Person p;
        p.set_name("abcd pqrs");
        p.set_id(123456);
        p.set_email("[email protected]");
        write(p);

        boost::asio::streambuf b;
        std::ostream os(&b);
        p.SerializeToOstream(&os);
        boost::asio::async_write(socket_, b,
                        boost::bind(&Server::handle_write, this,
                                boost::asio::placeholders::error));

In the client I am reading the message sent above using boost::asio::async_read. How do I find out the value of arg be set as an argument to boost::asio::transfer_at_least , in the code below?

 boost::asio::async_read(socket_, response_,
                            boost::asio::transfer_at_least(arg),
                            boost::bind(&Client::handle_read_header, this,
                                    boost::asio::placeholders::error));

Or else how do I make sure that boost::async_read returns after reading the entire object?

Answer 1

Correct, protobufs are not delimited. There's no knowing where a message ends from just a bytestream — even if you've seen all the fields you know about, maybe there are more repeated elements or somebody has extended the proto with a field you don't know about.

A common solution is to prefix frames with lengths (commonly encoded as VarInts). LevelDB and Szl both use this approach, for example. A VarInt can be unambiguously decoded byte by byte, and then you know how many more bytes to read before parsing your complete message.