Generic Parsing of PB in java

Question

Is it possible to parse protobuf in a generic fashion in Java?

I have looked into GeneratedMessage and could not find a way to parse any PB byte buffer into a GeneratedMessage.

Essentially, I am trying to parse a PB byte buffer into GeneratedMessage and then I would use reflection to detect fields inside it.

Answer 1

First of all, you can't parse PB data without knowing the schema. The schema originally comes from a ".proto" file and is typically embedded in the code generated by protoc. However, you can also tell protoc to store the schema in a format that's usable by the Java Protobuf library:

protoc --descriptor_set_out=mymessages.desc mymessages.proto

Then load it in your Java code:

FileInputStream fin = new FileInputStream("mymessages.desc");
Descriptors.FileDescriptorSet set =
  Descriptors.FileDescriptorSet.parseFrom(fin);
Descriptors.Descriptor md = set.getFile(0).getMessageType(0);

Once you have the schema for a message (Descriptor.Descriptor) parsing a message is easy:

byte[] data = ...;
DynamicMessage m = DynamicMessage.parseFrom(md, data);

DynamicMessage has a reflective API that lets you look through the fields.

The messy part is calling out to the protoc tool to convert the ".proto" file into a usable format. The C++ Protobuf library has a way to load ".proto" files directly, but unfortunately the Java Protobuf library does not.

Answer 2

You can use UnknownFieldSet to parse generic protobuf messages.

Then you can get individual fields using provided methods (e.g. asMap(), hasField(), getField())

E.g. (data taken from this question):

    byte[] msg = new byte[] { 0x0a, 0x10, 0x08, 0x7f, (byte)0x8a, 0x01, 0x04, 0x08, 0x02, 0x10, 0x03, (byte)0x92, 0x01, 0x04, 0x08, 0x02, 0x10, 0x03, 0x18, 0x01};
    UnknownFieldSet eee = UnknownFieldSet.parseFrom(msg);
    System.out.println(eee.toString());

Giving:

1: {
  1: 127
  17: {
    1: 2
    2: 3
  }
  18: {
    1: 2
    2: 3
  }
}
3: 1

Answer 3

This is right example:

private static DynamicMessage parseData(byte[] data) throws IOException, DescriptorValidationException {
    FileInputStream fin = new FileInputStream("test.desc");
    DescriptorProtos.FileDescriptorSet set = DescriptorProtos.FileDescriptorSet.parseFrom(fin);
    Descriptor md = Descriptors.FileDescriptor.buildFrom(set.getFile(0), new  Descriptors.FileDescriptor[] {}).findMessageTypeByName("Person");
    return DynamicMessage.parseFrom(md, data);
}

Answer 4

I have working solution tested with last protobuf v.3.1.0

This is upgraded solution raised on previous answers. Thanks to both authors.

import com.example.address.AddressBookManager;
import com.example.address.AddressBookProtos.AddressBook;
import com.google.protobuf.DescriptorProtos.FileDescriptorSet;
import com.google.protobuf.Descriptors;
import com.google.protobuf.Descriptors.Descriptor;
import com.google.protobuf.Descriptors.FileDescriptor;
import com.google.protobuf.DynamicMessage;

import java.io.File;
import java.io.InputStream;

public class DynamicMessageDemo {

    private static final String ADDRESS_BOOK_SOURCE_FILENAME = "test.ab";
    private static final String ADDRESS_BOOK_DESC_FILENAME =
        File.separator + "address.desc";

    public static void main(String[] args) throws Exception {

        InputStream is = DynamicMessageDemo.class
                .getResourceAsStream(ADDRESS_BOOK_DESC_FILENAME);

        FileDescriptorSet set = FileDescriptorSet.parseFrom(is);
        FileDescriptor fd = Descriptors.FileDescriptor.buildFrom(
                set.getFile(0), 
                new Descriptors.FileDescriptor[]{}
                );

        // "AddressBook" is the second message in my *.proto
        // so index must be '1'
        Descriptor messageType = fd.getMessageTypes().get(1);

        // for testing purpose 
        AddressBook book = AddressBookManager.readFromFile(ADDRESS_BOOK_SOURCE_FILENAME);
        byte[] data = book.toByteArray();

        DynamicMessage message = DynamicMessage.parseFrom(messageType, data);

        System.out.println("\n Dynamic message:\n" + message);
    }
}

Answer 5

Here is another way to generically parse a .desc file:

FileInputStream fin = new FileInputStream(descPath);
DescriptorProtos.FileDescriptorSet set = DescriptorProtos.FileDescriptorSet.parseFrom(fin);

List<FileDescriptor> dependencyFileDescriptorList = new ArrayList<>();
for(int i=0; i<set.getFileCount()-1;i++) {
    dependencyFileDescriptorList.add(Descriptors.FileDescriptor.buildFrom(set.getFile(i), new  Descriptors.FileDescriptor[] {}));
}
FileDescriptor desc = Descriptors.FileDescriptor.buildFrom(set.getFile(set.getFileCount()-1), dependencyFileDescriptorList.toArray(new FileDescriptor[0]));