Incremental write of Protocol Buffer object

Question

I have Protocol Buffer for logging data.

message Message {
    required double val1 = 1;
    optional int val2 = 2;
}

message BigObject {
    repeated Message message = 1;
}

I receive messages one per second. They stored in memory with my BigObject and they used for some tasks. But at the same time i want to store that messages in file for backup in case application crash. Simple writing BigObject every time will be waste of time. And I trying to find way to write only added messages since last write to file. Is there a way for that?

Answer 1

Protobuf is an appendable format, and your layout is ideal for this. Just open your file positioned at the end, and start with a new (empty) BigObject. Add/serialize just the new Message instance, and write to the file (from the end onwards).

Now, if you parse your file from the beginning you will get a single BigObject with all the Message instances (old and new).

You could actually do this by logging each individual Message as it arrives, as long as you wrap it in a BigObject each time, i.e. in pseudo-code

loop {
    msg = await NextMessage();
    wrapper = new BigObject();
    wrapper.Messages.Add(msg);

    file = OpenFileAtEnd();
    wrapper.WriteTo(file);
    file.Close();
}