Convert serialized protobuf output to python dictionary

Question

Given, a serialized protobuf (protocol buffer) output in the string format. I want to convert it to a python dictionary.

Suppose, this is the serialized protobuf, given as a python string:

person {
  info {
    name: John
    age: 20
    website: "https://mywebsite.com"
    eligible: True
  }
}

I want to convert the above python string to a python dictionary data, given as:

data = {
  "person": {
    "info": {
      "name": "John",
      "age": 20,
      "website": "https://mywebsite.com",
      "eligible": True,
    }
  }
}

I can write a python script to do the conversion, as follows:

• Append commas on every line not ending with curly brackets.
• Add an extra colon before the opening curly bracket.
• Surround every individual key and value pair with quotes.
• Finally, use the json.loads() method to convert it to a Python dictionary.

I wonder whether this conversion can be achieved using a simpler or a standard method, already available in protocol buffers. So, apart from the manual scripting using the steps I mentioned above, is there a better or a standard method available to convert the serialized protobuf output to a python dictionary?

Answer 1

You can use proto's Message class.

In [6]: import proto

In [7]: curr
Out[7]:
campaign {
  resource_name: "customers/1234/campaigns/5678"
  id: 9876
  name: "testing 1, 2, 3"
  advertising_channel_type: SEARCH
}
landing_page_view {
  resource_name: "customers/1234/landingPageViews/1234567890"
  unexpanded_final_url: "https://www.example.com/"
}

In [8]: proto.Message.to_dict(
   ...:     curr,
   ...:     use_integers_for_enums=False,
   ...:     including_default_value_fields=False,
   ...:     preserving_proto_field_name=True
   ...: )
Out[8]:
{'campaign': {'resource_name': 'customers/1234/campaigns/5678',
  'advertising_channel_type': 'SEARCH',
  'name': 'testing 1, 2, 3',
  'id': '9876'},
 'landing_page_view': {'resource_name': 'customers/1234/landingPageViews/1234567890',
  'unexpanded_final_url': 'https://www.example.com/'}}

Note that in to_dict, all kwargs default to True.

There's also a to_json method if you just want to immediately serialise the message without having to use json.dumps.

A caveat that's also worth noting is the proto package's recent memory leaks. The thread says that a fix has been issued but my experience when using it on larger datasets suggested otherwise 😅. Just because something works locally, doesn't mean that the container you deploy it to can handle the same load.

Answer 2

You can use google package MessageToDict function to covert proto3 message to python dict.

The relevant arguments are:

use_integers_for_enums: If true, print integers instead of enum names.
descriptor_pool: A Descriptor Pool for resolving types. If None use the
        default.
preserving_proto_field_name: If True, use the original proto field
        names as defined in the .proto file. If False, convert the field
        names to lowerCamelCase.
including_default_value_fields: If True, singular primitive fields,
        repeated fields, and map fields will always be serialized.  If
        False, only serialize non-empty fields.  Singular message fields
        and oneof fields are not affected by this option.

from google.protobuf.json_format import MessageToDict
import test_pb2

python_dict = MessageToDict(
    message,
    preserving_proto_field_name=True,
    including_default_value_fields=True,
    descriptor_pool=test_pb2.test_output)["output_list"]

where my proto3 file has test_output as output from module.