raw decoder for protobufs format

Question

I'd like to find a way to convert a binary protobuf message into a human readable description of the contained data, without using the .proto files.

The background is that I have a .proto message that it being rejected by the parser on Android, but it's not entirely clear why. I could go through the message by hand, but it's rather tedious.

I tried protoc --decode_raw, but it just gives the error "Failed to parse input.". I google hoping/expecting someone would have done a nice web utility that might do this, but haven't found anything obvious.

I'm just hoping to get some output like:

field 1: varint: 128 field 4: string: "foo" 

Any pointers in the right direction would be most welcome!

Answer 1

For Posterity: Google's protocol buffer tools have the ability to decode raw buffers.

Just send the unknown buffer to it and pass the--decode_raw flag

$ protoc --decode_raw < has_no_proto.buff 2 {   2: "Error retrieving information from server. [RH-02]" } 

So here's a message with field 2 set to an embedded message which in turn has its second field set to a string telling me I pissed off Google Play.

Type information isn't definite (it looks like it will try to display all binary data as strings -- but your requirement for varint/string/submessage distinction is met).

Answer 2

As noted in Michel de Ruiter's answer, it's possible that your protobuf message has a length-prefix. Assuming it does, this answer should help.

(NOTE: For most of the commands below, I'm assuming your protobuf message is stored in a file called input.)

protoc --decode_raw + dd for a single message:

If it's simply a single message, then you can indeed leverage protoc --decode_raw, but you need to strip off the length-prefix header first. Assuming the header is 4 bytes long you can use dd to strip the header off of input and then feed the output into protoc.

dd bs=1 skip=4 if=input 2>/dev/null | protoc --decode_raw 

protoc-decode-lenprefix --decode_raw for a single message:

I also wrote a script that handles the header stripping automatically:

protoc-decode-lenprefix --decode_raw < input 

This script is simply a wrapper on top of protoc --decode_raw, but handles parsing out the length-prefix header and then invoking protoc.

Now, this script isn't terribly useful in this case, because we can just use the dd trick above to strip the header off. However, say we have a data stream (e.g., a file or TCP stream) containing multiple messages that are framed with length-prefix headers....

protoc-decode-lenprefix --decode_raw for a stream of messages:

Instead of a single protobuf message in the input file, let's say input contained multiple protobuf messages which are framed by length-prefix headers. In this case it's not possible to just use the dd trick, because you need to actually read the contents of the length-prefix header to determine how long the subsequent message in the stream is, and thus how many bytes ahead the next header+message lies. So instead of worrying about all of that, you can simply use protoc-decode-lenprefix again:

protoc-decode-lenprefix --decode_raw < input 

protoc-decode-lenprefix --decode ... foo.proto for a stream of messages

This script can also be used to fully decode length-prefixed messages (instead of just "raw decode" them). It assumes you have access to the .proto files that define the protobuf message, just like the wrapped protoc command. The invocation syntax is identical to protoc --decode. For example, using the dd trick with protoc --decode, along with input being a Mesos task.info file, the syntax looks like this:

dd bs=1 skip=4 if=task.info 2>/dev/null | \ protoc --decode mesos.internal.Task \                       -I MESOS_CODE/src -I MESOS_CODE/include \                       MESOS_CODE/src/messages/messages.proto 

And the parameters are identical when using protoc-decode-lenprefix

cat task.info | \ protoc-decode-lenprefix --decode mesos.internal.Task \                       -I MESOS_CODE/src -I MESOS_CODE/include \                       MESOS_CODE/src/messages/messages.proto