exec_module() is not able to execute a module importing another module

Question

I have a structure of folders, similar to the one presented below:

main_folder
|
---> sub_folder_1
|    |
|    ---> main_script.py
|    |
|    ---> side_script.py
|
---> sub_folder_2
|    |
|    ---> main_script.py
|    |
|    ---> side_script.py
|
...

i.e. in each sub-folder I have files with the same names. It might happen that for some sub-folders main script imports some objects form the side script, simply by

import side_script

or

from side_script import *

Obviously, each main script refers to the side script in the same folder it is placed itself.

Now - in a totally different script I want to iterate over subfolders, load the main scripts, use some of their contents and save the results. Since in all sub-folders main files, which I want to import, are named in the same way, I need to closely watch which file I am importing (from which particular sub-folder), so instead of just adding values to the sys.path, I was suggested to use the importlib library and load the modules one by one, using the exec_module() method:

current_path = r'path\to\next\sub\folder'
spec = importlib.util.spec_from_file_location('main_script', os.path.join(current_path, 'main_script.py'))
executed_module = importlib.util.module_from_spec(spec)
spec.loader.exec_module(executed_module)

This works as long as main scripts do not import side scripts. If they do, I get the following error:

ModuleNotFoundError: No module named 'side_module'

Obviously, when I directly run any of the main scripts, there are no problems with that kind of import - they only appear when I import main scripts using exec_module() method. I suspect this is something related to the namespaces (exec_module() executes the module in a new namespace) but still, I don't know how I can make every main script I am executing as a module see the the scripts in the same sub-folder, without altering the main script itself.

Answer 1

As long as you got you __init__.py inside each sub_folder you can simply add the folder to the Python path. The Python path is what will be searched when you use import statements. E.g.:

import sys
sys.path.append(str(/full/path/to/sub_folder/))

I have not yet found a more elegant way to use importlib for this job or specify that the folder containing the module you're calling exec_module on is automaticaly added to the Python path. However, since you're manipulating the Python path this means that it can lead to unexpected behaviour when you return to the module that started the execution or start another module. Thus check this answer which shows an example implementation of how to reset the python path to it's original state after every call.