Serializing mutable state and sending it asynchronously over the network with nearly-zero-copy (Cap'n Proto + ZeroMQ)

Question

I have an application in which I'd like to send part of its mutable state over the network to another machine (there will be a cluster of those machines) to do some CPU-intensive computations on it and get back the results. Like asynchronous RPC. Such calls will happen many times during the execution of the program, so I'd like to make the overhead as small as possible, e.g. minimize the number of redundant copies of the data. The size of the data varies from tens of bytes to hundreds of KBs, maybe even few MBs. Its structure is relatively complex, it consists of a set of object trees, but the leaves contain only primitive types and the internal nodes contain minimal metadata.

I'm considering Cap'n Proto for serialization (though, in this case I'd have to create a redundant model for my data), and ZeroMQ for transport. On the client/main application side I'd like to use azmq, because I need Boost:Asio's features (namely coroutine/fiber support). The language is C++.

Summarizing with a very rough sketch:

RelativelyComplexState data;
CapnProtoRequest cp_req = buildRequest(data); // traverses my data, creates C'n P object
azmq_socket.async_send(boost::asio::buffer(cp_req, cp_req.size)); //azmq always copies the buffer? Not good.
// do other stuff while request is being processed remotely
// get notification from azmq/Boost:Asio when reply has arrived
azmq::message msg();
azmq_socket.async_receive(some_message_handler?); // get all the data into msg
CapnProtoResponse cp_resp = parseResponse(msg.cbuffer()); // interpret bytes as C'n P object, hopefully no copy
RelativelySimpleResult result = deserialize(cp_resp);

Is this feasible, or is there a better way? Would a schemaless serialization method (i.e. Boost::Serialization) make my life easier and/or the application more efficient in this case?

Also, what is the best way to send and receive a Cap'n Proto object with ZeroMQ/azmq, avoiding unnecessary copies? By looking at the source code of azmq, it seems that for sending, azmq always copies the buffer contents. What are the more subtle issues (segmenting/framing, etc.)? I'm not familiar with the libraries and haven't found any explanation or good examples.

Thank you!

Answer 1

I do not know much about ZeroMQ's interface but I can give advice on how to minimize copies from Cap'n Proto.

On the sending side, use capnp::MessageBuilder::getSegmentsForOutput() (capnp/message.h) to get direct pointers to the message's content without copying. This gives you an array of arrays of bytes (actually, words, but you can cast them to bytes). You need to somehow feed these to ZeroMQ without copying them. You'll need to make sure that the boundaries between segments are preserved -- the goal is to come up with exactly the same array of arrays on the receiving end. Maybe ZeroMQ has explicit support for multi-segment messages and can remember the segment boundaries for you; if not, you'll need to prefix your message with a table of segment sizes.

On the receiving side, once you have rebuilt your array of segments, construct a capnp::SegmentArrayMessageReader (capnp/message.h) and pass the array to the constructor. This will use the underlying data without copying. (Note that you will need to make sure that the data is aligned on a 64-bit boundary. I'm not sure if ZeroMQ guarantees this.)

Note that if both your client and server are C++, you may want to consider using Cap'n Proto's own RPC protocol, which is easier to set up and already avoids all unnecessary copies. However, integrating Cap'n Proto's event loop with boost::asio is currently non-trivial. It's possible -- for example you can look at node-capnp which integrates Cap'n Proto with libuv's event loop -- but may be more work than you want to do.

(Disclosure: I'm the author of Cap'n Proto.)