I have a python modules written in C, it has a main module and a submodule(name with a dot, not sure this can be called real submodule):
PyMODINIT_FUNC initsysipc(void) {
PyObject *module = Py_InitModule3("sysipc", ...);
...
init_sysipc_light();
}
static PyTypeObject FooType = { ... };
PyMODINIT_FUNC init_sysipc_light(void) {
PyObject *module = Py_InitModule3("sysipc.light", ...);
...
PyType_Ready(&FooType);
PyModule_AddObject(module, "FooType", &FooType);
}
The module is compiled as sysipc.so
, and when I put it in current directory, following import works without problem:
import sysipc
import sysipc.light
from sysipc.light import FooType
The problem is I want to put this module inside a namespace package, the folder structure is like this:
company/
company/__init__.py
company/dept/
company/dept/__init__.py
company/dept/sys/
company/dept/sys/__init__.py
company/dept/sys/sysipc.so
all the three __init__.py
just includes the standard setuptool
import line:
__path__ = __import__('pkgutil').extend_path(__path__, __name__)
in current directory, following imports does not work:
from company.dept.sys import sysipc;
from company.dept.sys.sysipc.light import FooType;
How should I import the types and methods defined in module sysipc.light
in this case?
===================================
Update with the actual error:
I have sysipc.so
built, if I run python in current directory as this module, import will work as expected:
[root@08649fea17ef 2]# python2
Python 2.7.18 (default, Jul 20 2020, 00:00:00)
[GCC 10.1.1 20200507 (Red Hat 10.1.1-1)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import sysipc
>>> import sysipc.light
>>>
If however if I put it into a namespace folder, like this:
company/
company/__init__.py
company/dept
company/dept/__init__.py
company/dept/sys
company/dept/sys/sysipc.so
company/dept/sys/__init__.py
import the submodule will not work:
>>> from company.dept.sys import sysipc
>>> from company.dept.sys import sysipc.light
File "<stdin>", line 1
from company.dept.sys import sysipc.light
^
SyntaxError: invalid syntax
>>> from company.dept.sys.sysipc import light
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
ImportError: cannot import name light
>>>
The module is built with this simple code, it is for python2. I also have same example for python3.
Python subpackages To access subpackages, we use the dot operator. This is the __init__.py file in the constants directory. We import the names tuple. This is the data.py module in the constants directory.
Usually in /lib/site-packages in your Python folder. (At least, on Windows.) You can use sys. path to find out what directories are searched for modules.
To make sure Python can always find the module.py , you need to: Place module.py in the folder where the program will execute. Include the folder that contains the module.py in the PYTHONPATH environment variable. Or you can place the module.py in one of the folders included in the PYTHONPATH variable.
Quoting from https://www.python.org/dev/peps/pep-0489/#multiple-modules-in-one-library :
To support multiple Python modules in one shared library, the library can export additional PyInit* symbols besides the one that corresponds to the library's filename.
Note that this mechanism can currently only be used to load extra modules, but not to find them. (This is a limitation of the loader mechanism, which this PEP does not try to modify.) ...
In other words, you need to restructure the project as follows for importlib
to be able to find the submodule light
in the sysipc
package:
company/__init__.py
company/dept/__init__.py
company/dept/sys/__init__.py
company/dept/sys/sysipc/__init__.py
company/dept/sys/sysipc/sysipc.so
company/dept/sys/sysipc/light.so -> sysipc.so # hardlink
The hardlink between light.so
and sysipc.so
can be created with:
ln company/dept/sys/sysipc/sysipc.so company/dept/sys/sysipc/light.so
Then in company/dept/sys/sysipc/__init__.py
you import all symbols from sysipc.so
using:
from .sysipc import *
In addition, you need to change the name of the submodule C extension init function from init_sysipc_light
to init_light
for Python2, or from PyInit_sysipc_light
to PyInit_light
for Python3, since importlib
loads modules by looking for an exported PyInit_<module name>
from the dynamic module and the module name here is only light
, i.e., the parent package prefix is not part of the (sub)module name.
Here is the extension code (Python3) and a couple of functions for testing:
#include <Python.h>
PyObject *sysipc_light_foo(PyObject *self, PyObject *args) {
printf("[*] sysipc.light.foo\n");
return PyLong_FromLong(0);
}
static PyMethodDef sysipc_light_methods[] = {
{"foo", (PyCFunction)sysipc_light_foo, METH_VARARGS, "sysipc.light.foo function"},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef sysipc_light_module = {
PyModuleDef_HEAD_INIT,
"sysipc.light",
"sysipc child module",
-1,
sysipc_light_methods
};
PyMODINIT_FUNC PyInit_light(void)
{
PyObject *module = NULL;
module = PyModule_Create(&sysipc_light_module);
return module;
}
PyObject *sysipc_bar(PyObject *self, PyObject *args) {
printf("[*] sysipc.bar\n");
return PyLong_FromLong(0);
}
static PyMethodDef sysipc_methods[] = {
{"bar", (PyCFunction)sysipc_bar, METH_VARARGS, "sysipc.bar function"},
{NULL, NULL, 0, NULL}
};
static struct PyModuleDef sysipc_module = {
PyModuleDef_HEAD_INIT,
"sysipc",
"sysipc parent module",
-1,
sysipc_methods
};
PyMODINIT_FUNC PyInit_sysipc(void)
{
PyObject *module = NULL;
module = PyModule_Create(&sysipc_module);
PyInit_light();
return module;
}
test.py:
#!/usr/bin/env python3
from company.dept.sys import sysipc
from company.dept.sys.sysipc import light
sysipc.bar()
light.foo()
Output:
[*] sysipc.bar
[*] sysipc.light.foo
There are two issues here: first, Py_InitModule
and friends expect to create the module being imported. The hint is that the string you pass it is not the fully-qualified name of the module: Python uses the name it already knows to determine where in sys.modules
to put the new object. You can, however, use the fully qualified name; other magic attributes like __file__
will have the correct values.
The second issue is that the attribute light
needs to be set on the containing module for from
imports to work.
Meanwhile, there's no reason to have a separate initialization function (that the interpreter will never call), and combining them avoids the need to recover a pointer to the module later:
static PyTypeObject FooType = { ... };
PyMODINIT_FUNC initsysipc(void) {
PyObject *module = Py_InitModule3("sysipc", ...);
...
PyObject *const sub = Py_InitModule3("company.dept.sys.sysipc.light", ...);
...
PyType_Ready(&FooType);
// PyModule_AddObject steals a reference:
Py_INCREF(FooType);
PyModule_AddObject(sub, "FooType", &FooType);
Py_INCREF(sub);
PyModule_AddObject(module, "light", sub);
}
That said, sysipc
will still not be a proper package: at the least, it lacks __path__
. If that matters, you might prefer MEE's answer that uses a real (if more complicated) package architecture.
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With