Errors while dynamic imports using importlib in python3

Question

I have been trying to use importlib with python3 (3.6).

Directory structure

main.py

#Note: I will only modify line 4 that uses importlib
import importlib
if __name__ == '__main__':
    print("In main.py")
    hello = importlib.import_module('hello', package='./')
    print("Loaded hello.py")
    hello.hello()

hello.py

def hello():
    print('Hello world')

folder/hello.py

def hello():
    print('Hello world in folder')

Observations

If I do

hello = importlib.import_module('hello', package='./') or

hello = importlib.import_module('hello')

It imports hello.py from the root folder and prints hello world.

If I do

hello = importlib.import_module('folder.hello')

It imports folder/hello.py from the root folder and prints hello world in folder.

But if I do

hello = importlib.import_module('hello', package='folder') or

hello = importlib.import_module('hello', package='./folder')

It gives error

Traceback (most recent call last):
  File "main.py", line 4, in <module>
    hello = importlib.import_module('hello', package='./folder')
  File "/usr/lib/python3.6/importlib/__init__.py", line 126, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 994, in _gcd_import
  File "<frozen importlib._bootstrap>", line 971, in _find_and_load
  File "<frozen importlib._bootstrap>", line 953, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'hello'

Problem

I am not sure what is going on here. I am pretty sure there is something wrong with my understanding of python modules and packages. Can someone explain why this is the expected behavior?

Answer 1

If the first argument, the module to be imported is an absolute module reference ( has no leading .), the seond argument is completely ignored.

To import a module a relative to another module b, you have to use

a = importlib.import_module('.a', package='b')

In your case, this should work

hello = importlib.import_module('.hello', package='folder')

As a rule of thumb, import package should work if you want to use package as second argument.

from package import module

then becomes

importlib.import_module(module, package)

Answer 2

@Mahesh's answer is 100% correct and spot-on, but I guess we need to go one level deep to make you understand it better

Below is the code for import_module

def import_module(name, package=None):
    """Import a module.

    The 'package' argument is required when performing a relative import. It
    specifies the package to use as the anchor point from which to resolve the
    relative import to an absolute import.

    """
    level = 0
    if name.startswith('.'):
        if not package:
            msg = ("the 'package' argument is required to perform a relative "
                   "import for {!r}")
            raise TypeError(msg.format(name))
        for character in name:
            if character != '.':
                break
            level += 1
    return _bootstrap._gcd_import(name[level:], package, level)

You can see if the name doesn't start with . then if part doesn't get executed. You just have return _bootstrap._gcd_import(name[level:], package, level) which gets executed with level=0 as the value

Now let's get into that function, which has below code

def _gcd_import(name, package=None, level=0):
    """Import and return the module based on its name, the package the call is
    being made from, and the level adjustment.

    This function represents the greatest common denominator of functionality
    between import_module and __import__. This includes setting __package__ if
    the loader did not.

    """
    _sanity_check(name, package, level)
    if level > 0:
        name = _resolve_name(name, package, level)
    return _find_and_load(name, _gcd_import)

Again in this it just executes _find_and_load(name, _gcd_import), now because level is 0 from our previous code, the package parameter is not being passed or used at all by _find_and_load method. Now you can easily verify this by running below

import importlib
hello = importlib.import_module('hello', package='IAmNotAfolder')
hello.hello()

And it will print Hello World from the base hello.py

So as you can see the package parameter is not used at all when the name doesn't start with ., which is for relative imports. That is why you get an error No module named 'hello' because it is trying to import hello.py from base folder irrespective of what you have in package.

Hope this answer make it easier for you to understand what is happening behind the scenes