Relative imports with ProtoBuf: Generating Python classes with ProtoBuf gives ModuleNotFoundError

Question

Does protobuf support relative imports for python?

I have been unsuccessful in creating a protobuf build script that supports this. When generating python-classes from my .proto-files, I am only able to import the python-modules if I launch python from the same folder as where the generated .py-files were created.

I've constructed the following MVP. Ideally, I would like a structure where the generated python code it placed in a separate folder (e.g. ./generated) that I can then move into other projects. I've posted the approaches that I have gotten to work, but I am hoping someone more experienced will be able to point me towards a better solution.

General info:

• Python 3.6.8
• protobuf 3.11.3

Folder structure:

.
|--- schemas
     |---- main_class.proto
     |---- sub
           |----sub_class.proto
|--- generated

Attempt 1: Relative imports

main_class.proto:

syntax = "proto3";

import public "sub/sub_class.proto";

message MainClass {
    repeated SubClass subclass = 1;
}

sub_class.proto:

syntax = "proto3";

message LogMessage {
    enum Status {
        STATUS_ERROR = 0;
        STATUS_OK = 1;
    }

    Status status = 1;
    string timestamp = 2;
}

message SubClass {
    string name = 1;
    repeated LogMessage log = 2;
}

Protoc command:

From the root folder:

protoc -I=schemas --python_out=generated main_class.proto sub/sub_class.proto

This puts the python-files in the ./generated folder.

What works and what doesn't

Using the approach above, I am able to launch python in the folder ./generated and import using

import main_class_pb2 as MC_proto.

However, when I launch python from the . root folder (or any other folder for that matter), importing using

import generated.main_class_pb2 as MC_proto

yields the error ModuleNotFoundError: No module named 'sub'. Based on this post, I manually modified the generated main_class_pb2.py-file as follows

# Original
# from sub import sub_class_pb2 as sub_dot_sub__class__pb2
# from sub.sub_class_pb2 import *

# Fix
from .sub import sub_class_pb2 as sub_dot_sub__class__pb2
from .sub.sub_class_pb2 import *

By adding the . in at the beginning of the import statement, I am now able to import the module from the root folder using import generated.main_class_pb2 as MC_proto. However, it is very impractical having to edit the generated files manually every time, so I don't like this approach.

Attempt 2: Absolute imports

My second approach was to try absolute imports. If I knew where my project root folder would be, I could then move the .proto-files to where I wanted the python-classes to be and generate them there. For this example, I used the same folder structure as before, but without the ./generated-folder. I also had to change the root folder for the protoc command, which required me to modify the import statement in the main_class.proto file, as follows:

main_class.proto:

syntax = "proto3";

// Old
//import public "sub/sub_class.proto";
// New
import public "schemas/sub/sub_class.proto";

message MainClass {
    repeated SubClass subclass = 1;
}

Protoc command

protoc -I=. --python_out=. schemas/main_class.proto schemas/sub/sub_class.proto

What works and what doesn't

Assuming my root folder is also my project's root folder, this approach now lets me launch python in the root folder and import the module using

import schemas.main_class_pb2

However, this means that my .proto-files must be located in the same folders as my python-files in that project, which seems pretty messy. It also means that you must generate the python-files from the same root folder as the project, which is not always possible. The .proto-files might be used to create a common interface for two totally different applications, and having to maintain two slightly different protobuf-projects seems to defeat the purpose of using protobuf.

---

Example python code

I am providing some sample python code that can be used to test that the import works and that the classes work as intended. This example is from attempt 1, and assumes that python is launched from the ./generated folder

import main_class_pb2 as MC_proto

sub1, sub2 = (MC_proto.SubClass(name='sub1'),
              MC_proto.SubClass(name='sub2'))

sub1.log.append(MC_proto.LogMessage(status=1, timestamp='2020-01-01'))
sub1.log.append(MC_proto.LogMessage(status=0, timestamp='2020-01-01'))
sub2.log.append(MC_proto.LogMessage(status=1, timestamp='2020-01-02'))

main = MC_proto.MainClass(subclass=[sub1, sub2])
main
Out[]: 
subclass {
  name: "sub1"
  log {
    status: STATUS_OK
    timestamp: "2020-01-01"
  }
  log {
    timestamp: "2020-01-01"
  }
}
subclass {
  name: "sub2"
  log {
    status: STATUS_OK
    timestamp: "2020-01-02"
  }
}

Answer 1

There is no way to tell protoc to use relative imports when generating Python code. Checking the protoc source code in C++ it's clear it only works with absolute imports. Take a look below:

src/google/protobuf/compiler/python/generator.cc -> The piece of code that generates the imports section of other proto files

// Prints Python imports for all modules imported by |file|.
void Generator::PrintImports() const {
  for (int i = 0; i < file_->dependency_count(); ++i) {
    const std::string& filename = file_->dependency(i)->name();

    std::string module_name = ModuleName(filename);
    std::string module_alias = ModuleAlias(filename);
    if (ContainsPythonKeyword(module_name)) {
      // If the module path contains a Python keyword, we have to quote the
      // module name and import it using importlib. Otherwise the usual kind of
      // import statement would result in a syntax error from the presence of
      // the keyword.
      printer_->Print("import importlib\n");
      printer_->Print("$alias$ = importlib.import_module('$name$')\n", "alias",
                      module_alias, "name", module_name);
    } else {
      int last_dot_pos = module_name.rfind('.');
      std::string import_statement;
      if (last_dot_pos == std::string::npos) {
        // NOTE(petya): this is not tested as it would require a protocol buffer
        // outside of any package, and I don't think that is easily achievable.
        import_statement = "import " + module_name;
      } else {
        import_statement = "from " + module_name.substr(0, last_dot_pos) +
                           " import " + module_name.substr(last_dot_pos + 1);
      }
      printer_->Print("$statement$ as $alias$\n", "statement", import_statement,
                      "alias", module_alias);
    }

    CopyPublicDependenciesAliases(module_alias, file_->dependency(i));
  }
  printer_->Print("\n");

  // Print public imports.
  for (int i = 0; i < file_->public_dependency_count(); ++i) {
    std::string module_name = ModuleName(file_->public_dependency(i)->name());
    printer_->Print("from $module$ import *\n", "module", module_name);
  }
  printer_->Print("\n");
}

This function uses the module_name to generate the following snippet in your 1st attempt:

from sub import sub_class_pb2 as sub_dot_sub__class__pb2

from sub.sub_class_pb2 import *

And module_name comes from the function ModuleName below:

// Returns the Python module name expected for a given .proto filename.
std::string ModuleName(const std::string& filename) {
  std::string basename = StripProto(filename);
  ReplaceCharacters(&basename, "-", '_');
  ReplaceCharacters(&basename, "/", '.');
  return basename + "_pb2";
}

As you can see there is no flag or logic in this function to generate relative imports.

IMO, I think the best approach is to use your 2nd attempt but on a different package and then you can import it from your code in Python.