Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How do you tell pylint what the members of a protobuf-generated object are?

I would like to release a python package for a set of protobuf messages. The protobuf compiler (protoc) generates a python library that does not actually define types/classes the typical sense, but rather dynamically constructs them. Is there any way to hint to pylint what the members and fields of these classes are?

For example, consider the following simple protobuf message specification:

message Person {
  required string name = 1;
  required int32 id = 2;
  optional string email = 3;
}

The compiler generates the following long pile of code:

# Generated by the protocol buffer compiler.  DO NOT EDIT!
# source: test.proto

import sys
_b=sys.version_info[0]<3 and (lambda x:x) or (lambda x:x.encode('latin1'))
from google.protobuf import descriptor as _descriptor
from google.protobuf import message as _message
from google.protobuf import reflection as _reflection
from google.protobuf import symbol_database as _symbol_database
from google.protobuf import descriptor_pb2
# @@protoc_insertion_point(imports)

_sym_db = _symbol_database.Default()




DESCRIPTOR = _descriptor.FileDescriptor(
  name='test.proto',
  package='',
  serialized_pb=_b('\n\ntest.proto\"1\n\x06Person\x12\x0c\n\x04name\x18\x01 \x02(\t\x12\n\n\x02id\x18\x02 \x02(\x05\x12\r\n\x05\x65mail\x18\x03 \x01(\t')
)
_sym_db.RegisterFileDescriptor(DESCRIPTOR)




_PERSON = _descriptor.Descriptor(
  name='Person',
  full_name='Person',
  filename=None,
  file=DESCRIPTOR,
  containing_type=None,
  fields=[
    _descriptor.FieldDescriptor(
      name='name', full_name='Person.name', index=0,
      number=1, type=9, cpp_type=9, label=2,
      has_default_value=False, default_value=_b("").decode('utf-8'),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      options=None),
    _descriptor.FieldDescriptor(
      name='id', full_name='Person.id', index=1,
      number=2, type=5, cpp_type=1, label=2,
      has_default_value=False, default_value=0,
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      options=None),
    _descriptor.FieldDescriptor(
      name='email', full_name='Person.email', index=2,
      number=3, type=9, cpp_type=9, label=1,
      has_default_value=False, default_value=_b("").decode('utf-8'),
      message_type=None, enum_type=None, containing_type=None,
      is_extension=False, extension_scope=None,
      options=None),
  ],
  extensions=[
  ],
  nested_types=[],
  enum_types=[
  ],
  options=None,
  is_extendable=False,
  extension_ranges=[],
  oneofs=[
  ],
  serialized_start=14,
  serialized_end=63,
)

DESCRIPTOR.message_types_by_name['Person'] = _PERSON

Person = _reflection.GeneratedProtocolMessageType('Person', (_message.Message,), dict(
  DESCRIPTOR = _PERSON,
  __module__ = 'test_pb2'
  # @@protoc_insertion_point(class_scope:Person)
  ))
_sym_db.RegisterMessage(Person)


# @@protoc_insertion_point(module_scope)
like image 392
Arun Chaganty Avatar asked Apr 28 '17 07:04

Arun Chaganty


People also ask

How do I read a proto file in Python?

import sys import myprotocol_pb2 as proto import varint # (this is the varint.py file) data = open("filename. bin", "rb"). read() # read file as string decoder = varint. decodeVarint32 # get a varint32 decoder # others are available in varint.py next_pos, pos = 0, 0 while pos < len(data): msg = proto.

What is a Protobuf message?

Protocol Buffers (Protobuf) is a free and open-source cross-platform data format used to serialize structured data. It is useful in developing programs to communicate with each other over a network or for storing data.

How does Google Protobuf work?

The Protobuf is a binary transfer format, meaning the data is transmitted as a binary. This improves the speed of transmission more than the raw string because it takes less space and bandwidth. Since the data is compressed, the CPU usage will also be less.

What is protocol buffer in Python?

Protocol buffers are Google's language-neutral, platform-neutral, extensible mechanism for serializing structured data – think XML, but smaller, faster, and simpler. You define how you want your data to be structured once … The ProtoBuf interface describes the structure of the data to be sent.


1 Answers

There didn't seem to be any support for automatically hinting to Pylint the names of the fields on protobuf messages, so I cobbled together a small extension to Pylint to statically determine these.

Introducing pylint-protobuf:

from example_pb2 import Person
p = Person()
p.invalid_field = 123

Usage:

$ pip install pylint-protobuf
$ pylint --load-plugins=pylint_protobuf example.py
************* Module example
E:  3, 0: Field 'invalid_field' does not appear in the declared
fields of protobuf-generated class 'Person' and will raise
AttributeError on access (protobuf-undefined-attribute)

Fair warning: it's pretty alpha, features not supported (yet!) include nested imports and repeated fields.

like image 57
nelfin Avatar answered Sep 28 '22 03:09

nelfin