Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Storing multiple messages in one protocol buffer binary file

I have repeating messages which I want to store in a single file. Currently I have to wrap this repeating message in another message. Is there a way around this?

package foo;

message Box {
  required int32 tl_x = 1;
  required int32 tl_y = 2;
  required int32 w = 3;
  required int32 h = 4;
}

message Boxes {
  repeated Box boxes = 1;
}
like image 551
Dat Chu Avatar asked Apr 07 '11 19:04

Dat Chu


People also ask

Is Protocol a buffer binary?

Protocol buffers, or Protobuf, is a binary format created by Google to serialize data between different services. Google made this protocol open source and now it provides support, out of the box, to the most common languages, like JavaScript, Java, C#, Ruby and others.

Is protobuf compressed?

Protobuf achieves good compression ratios through varint encoding. It might not be necessary to add an extra layer of compression in most cases.

What are protocol buffers used for?

Protocol buffers provide a language-neutral, platform-neutral, extensible mechanism for serializing structured data in a forward-compatible and backward-compatible way. It's like JSON, except it's smaller and faster, and it generates native language bindings.

Why is protobuf faster?

The Protobuf is a binary transfer format, meaning the data is transmitted as a binary. This improves the speed of transmission more than the raw string because it takes less space and bandwidth. Since the data is compressed, the CPU usage will also be less.


3 Answers

Here's what "Techniques" section of the Protocol Buffers documentation says about repeated messages:

If you want to write multiple messages to a single file or stream, it is up to you to keep track of where one message ends and the next begins. The Protocol Buffer wire format is not self-delimiting, so protocol buffer parsers cannot determine where a message ends on their own. The easiest way to solve this problem is to write the size of each message before you write the message itself. When you read the messages back in, you read the size, then read the bytes into a separate buffer, then parse from that buffer. (If you want to avoid copying bytes to a separate buffer, check out the CodedInputStream class (in both C++ and Java) which can be told to limit reads to a certain number of bytes.)

There's also a conventional way of implementing this in C++ and Java. Take a look at this Stack Overflow thread for details: Are there C++ equivalents for the Protocol Buffers delimited I/O functions in Java?

like image 55
alavrik Avatar answered Sep 21 '22 15:09

alavrik


Protobuf doesn't support this functionality. It can be used to just serialize one message, but this serialized message doesn't contain information about its type (Box or Boxes) and length. So if you want to store multiple message you have to include type and length of message as well. Writing algorithm (in pseudo language) could look like this:

for every message {
    write(type_of_message) // 1 byte long
    write(length_of_serialized_message) // 4 bytes long
    write(serialized_message)
}

Load algorithm:

while(end_of_file) {

    type = read(1) // 1 byte
    length = read(4) // 4 bytes
    buffer = read(length)
    switch (type) {
      case 1:
         deserialise_message_1(buffer)
      case 2:
         deserialise_message_2(buffer)
    }
}
like image 45
Zuljin Avatar answered Sep 20 '22 15:09

Zuljin


I was just working on this problem and ended up going with Parquet. Parquet works perfectly for storing a bunch of Protobuf messages in a file and makes it easier to work with them later on.

This bit of code will create the Parquet file:

Path path = new Path("/tmp/mydata.parq");
CompressionCodecName codecName = CompressionCodecName.SNAPPY;
int blockSize = 134217728;
int pageSize = 1048576;
boolean enableDictionary = true;
boolean validating = false;

ProtoParquetWriter<Message> writer
    = new ProtoParquetWriter<>(
        path,
        Box.class,
        codecName,
        blockSize,
        pageSize,
        enableDictionary,
        validating
    );

for (Message message : messages) {
    writer.write(message);
}

writer.close();

It might not suit your use case but I thought that it was worth a mention here.

like image 25
Collin Krawll Avatar answered Sep 20 '22 15:09

Collin Krawll