Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

python import path for sub modules if put in namespace package

I have a python modules written in C, it has a main module and a submodule(name with a dot, not sure this can be called real submodule):

PyMODINIT_FUNC initsysipc(void) {
    PyObject *module = Py_InitModule3("sysipc", ...);
    ...
    init_sysipc_light();
}

static PyTypeObject FooType = { ... };
PyMODINIT_FUNC init_sysipc_light(void) {
    PyObject *module = Py_InitModule3("sysipc.light", ...);
    ...
    PyType_Ready(&FooType);
    PyModule_AddObject(module, "FooType", &FooType);
}

The module is compiled as sysipc.so, and when I put it in current directory, following import works without problem:

import sysipc
import sysipc.light
from sysipc.light import FooType

The problem is I want to put this module inside a namespace package, the folder structure is like this:

company/
company/__init__.py
company/dept/
company/dept/__init__.py
company/dept/sys/
company/dept/sys/__init__.py
company/dept/sys/sysipc.so

all the three __init__.py just includes the standard setuptool import line:

__path__ = __import__('pkgutil').extend_path(__path__, __name__)

in current directory, following imports does not work:

from company.dept.sys import sysipc;
from company.dept.sys.sysipc.light import FooType;

How should I import the types and methods defined in module sysipc.light in this case?

===================================

Update with the actual error:

I have sysipc.so built, if I run python in current directory as this module, import will work as expected:

[root@08649fea17ef 2]# python2
Python 2.7.18 (default, Jul 20 2020, 00:00:00)
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysipc
>>> import sysipc.light
>>>

If however if I put it into a namespace folder, like this:

company/
company/__init__.py
company/dept
company/dept/__init__.py
company/dept/sys
company/dept/sys/sysipc.so
company/dept/sys/__init__.py

import the submodule will not work:

>>> from company.dept.sys import sysipc
>>> from company.dept.sys import sysipc.light
  File "<stdin>", line 1
    from company.dept.sys import sysipc.light
                                   ^
SyntaxError: invalid syntax
>>> from company.dept.sys.sysipc import light
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name light
>>>

The module is built with this simple code, it is for python2. I also have same example for python3.

like image 248
fluter Avatar asked Sep 20 '20 12:09

fluter


People also ask

How do you access sub packages in Python?

Python subpackages To access subpackages, we use the dot operator. This is the __init__.py file in the constants directory. We import the names tuple. This is the data.py module in the constants directory.

Where are imported modules stored in Python?

Usually in /lib/site-packages in your Python folder. (At least, on Windows.) You can use sys. path to find out what directories are searched for modules.

How do you add a path to a Python module?

To make sure Python can always find the module.py , you need to: Place module.py in the folder where the program will execute. Include the folder that contains the module.py in the PYTHONPATH environment variable. Or you can place the module.py in one of the folders included in the PYTHONPATH variable.


2 Answers

Quoting from https://www.python.org/dev/peps/pep-0489/#multiple-modules-in-one-library :

To support multiple Python modules in one shared library, the library can export additional PyInit* symbols besides the one that corresponds to the library's filename.

Note that this mechanism can currently only be used to load extra modules, but not to find them. (This is a limitation of the loader mechanism, which this PEP does not try to modify.) ...

In other words, you need to restructure the project as follows for importlib to be able to find the submodule light in the sysipc package:

company/__init__.py
company/dept/__init__.py
company/dept/sys/__init__.py
company/dept/sys/sysipc/__init__.py
company/dept/sys/sysipc/sysipc.so
company/dept/sys/sysipc/light.so -> sysipc.so  # hardlink

The hardlink between light.so and sysipc.so can be created with:

ln company/dept/sys/sysipc/sysipc.so company/dept/sys/sysipc/light.so

Then in company/dept/sys/sysipc/__init__.py you import all symbols from sysipc.so using:

from .sysipc import *

In addition, you need to change the name of the submodule C extension init function from init_sysipc_light to init_light for Python2, or from PyInit_sysipc_light to PyInit_light for Python3, since importlib loads modules by looking for an exported PyInit_<module name> from the dynamic module and the module name here is only light, i.e., the parent package prefix is not part of the (sub)module name.

Here is the extension code (Python3) and a couple of functions for testing:

#include <Python.h>

PyObject *sysipc_light_foo(PyObject *self, PyObject *args) {
  printf("[*] sysipc.light.foo\n");
  return PyLong_FromLong(0);
}

static PyMethodDef sysipc_light_methods[] = {
    {"foo", (PyCFunction)sysipc_light_foo, METH_VARARGS, "sysipc.light.foo function"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef sysipc_light_module = {
    PyModuleDef_HEAD_INIT,
    "sysipc.light",
    "sysipc child module",
    -1,
    sysipc_light_methods
};

PyMODINIT_FUNC PyInit_light(void)
{
    PyObject *module = NULL;

    module = PyModule_Create(&sysipc_light_module);

    return module;
}

PyObject *sysipc_bar(PyObject *self, PyObject *args) {
  printf("[*] sysipc.bar\n");
  return PyLong_FromLong(0);
}

static PyMethodDef sysipc_methods[] = {
    {"bar", (PyCFunction)sysipc_bar, METH_VARARGS, "sysipc.bar function"},
    {NULL, NULL, 0, NULL}
};

static struct PyModuleDef sysipc_module = {
    PyModuleDef_HEAD_INIT,
    "sysipc",
    "sysipc parent module",
    -1,
    sysipc_methods
};

PyMODINIT_FUNC PyInit_sysipc(void)
{
    PyObject *module = NULL;

    module = PyModule_Create(&sysipc_module);

    PyInit_light();

    return module;
}

test.py:

#!/usr/bin/env python3

from company.dept.sys import sysipc
from company.dept.sys.sysipc import light

sysipc.bar() 
light.foo()

Output:

[*] sysipc.bar
[*] sysipc.light.foo
like image 84
MEE Avatar answered Oct 19 '22 23:10

MEE


There are two issues here: first, Py_InitModule and friends expect to create the module being imported. The hint is that the string you pass it is not the fully-qualified name of the module: Python uses the name it already knows to determine where in sys.modules to put the new object. You can, however, use the fully qualified name; other magic attributes like __file__ will have the correct values.

The second issue is that the attribute light needs to be set on the containing module for from imports to work.

Meanwhile, there's no reason to have a separate initialization function (that the interpreter will never call), and combining them avoids the need to recover a pointer to the module later:

static PyTypeObject FooType = { ... };
PyMODINIT_FUNC initsysipc(void) {
    PyObject *module = Py_InitModule3("sysipc", ...);
    ...
    PyObject *const sub = Py_InitModule3("company.dept.sys.sysipc.light", ...);
    ...
    PyType_Ready(&FooType);
    // PyModule_AddObject steals a reference:
    Py_INCREF(FooType);
    PyModule_AddObject(sub, "FooType", &FooType);
    Py_INCREF(sub);
    PyModule_AddObject(module, "light", sub);
}

That said, sysipc will still not be a proper package: at the least, it lacks __path__. If that matters, you might prefer MEE's answer that uses a real (if more complicated) package architecture.

like image 20
Davis Herring Avatar answered Oct 19 '22 21:10

Davis Herring