Python import mechanism and module mocks

Question

This is more an effort to understand how Python (3.9 in this instance) works than an effort to solve an actual problem so please bear with me and disregard the nonsensical way of m3. I just wanted to replicate something I'm dealing with.

I have the following structure:

├── m1.py
└── m2
    └── m3
        ├── __init__.py
        └── m3.py

m2/m3/init.py:

from .m3 import *

m2/m3/m3.py:

def m3func():
    print('m3 func is here')

From now on I will be making changes to m1.py

This is working and I was expecting it to work:

import m2.m3
m2.m3.m3func()

This is not failing so it replaced the module for the Mock. I was also expecting this to work the way it does.

import sys
from unittest.mock import Mock
sys.modules['m2.m3'] = Mock()
import m2.m3 as alias
alias.m3func()

Same for this

import sys
from unittest.mock import Mock
sys.modules['m2.m3'] = Mock()
from m2 import m3
m3.m3func()

I don't understand what's happening here:

import sys
from unittest.mock import Mock
sys.modules['m2.m3'] = Mock()
import m2.m3
m2.m3.m3func()

m2.m3.m3func()
AttributeError: module 'm2' has no attribute 'm3'

What are the differences between import m2.m3, from m2 import m3 and import m2.m3 as alias What else am I not understanding and is there a way to fix the last version so that it won't throw the AttributeError? Im my example m2 is empty but in actuality, I don't want to swap it out entirely because it does contain things that I care about. I would just like to target m3. Is there a recommendation as far as best practices go to using code like this: m2.m3.m3func()?

Answer 1

You should definitely read the official documentation, if you haven't yet.

Whenever you use an import statement, the following happens:

1. Python searches the module, and processes newly discovered modules found along the way
2. Python introduces one or multiple variables to use whatever was imported

Discovery Problems

Python uses sys.modules internally for searching modules, and updates its entries as it finds modules and parent modules along the way. So, when you import m2.m3.m3, it adds entries with keys 'm2', 'm2.m3', 'm2.m3.m3' to store objects referencing those modules. You can use the following function to debug sys.modules before/after every statement.

def print_status_relevant_modules():
    import sys
    print(sys.modules.get("m2", "<m2 not loaded>"))
    print(sys.modules.get("m2.m3", "<m2.m3 not loaded>"))
    print(sys.modules.get("m2.m3.m3", "<m2.m3.m3 not loaded>"))
    print()

Whenever Python discovers a new package (a directory with a __init__.py file in it) or a module.py module, it will also process that module. In this case, when Python discovers the existence of the module m2.m3, it will execute the contents of m2/m3/__init__.py, and thus execute the import statement from .m3 import *, resulting in the discovery of the module m2.m3.m3 (and the introduction of the local variable m3func to m2.m3).

Your problems start when start mocking the entry with key "m2.m3" in sys.modules, because now you have disrupted Python's process of searching for modules. Because you mocked the entry for the module m2.m3 in sys.modules beforehand, Python thinks that it has already processed this module, so Python will never execute its __init__.py file. As a result, m2.m3.m3 will never be discovered, no entry will be added, and the local variable for m3func will never be introduced to m2.m3.

If you wonder why you are not seeing any errors when mocking the module's entry, even though you are calling the m3func(), it's because mocks will accept any calls, expecting you to later verify that a certain call was made.

Different Import Statements

The big difference between all the different import statements is which local variables are introduced:

• import m2.m3 results in a local variable m2 with attribute m3; the variable that is introduced refers to a Module instance representing the module m2
• from m2 import m3 results in a local variable m3; the variable that is introduced refers to a Module instance representing the module m2.m3
• import m2.m3 as alias results in a local variable alias; the variable that is introduced refers to a Module instance representing the module m2.m3

You can use the statement print(dir()) anywhere to see which local variables are defined, or on objects.

print(dir())    # m2 is not defined
import m2.m3
print(dir())    # m2 is defined
print(dir(m2))  # shows that m2 has an attribute m3

As an added bonus, the statement from .m3 import * in m2/m3/__init__.py results in all local variables of m2.m3.m3 to be imported to m2.m3. In this case, only the variable m3func is added.

When you use the import statement import m2.m3, the local variable m2 is introduced, which refers to a Module instance representing the module m2. While searching for modules, Python should've discovered the module m2.m3, and added the attribute m3 to the Module instance representing the module m2, to refer to the Module instance representing the module m2.m3. However, because you mocked sys.modules['m2.m3'] beforehand, the module m2.m3 is never discovered, and thus the attribute m3 is never added to the Module instance representing module m2. This eventually results in an error, when you try to access m2.m3.

When you use the import statement import m2.m3 as alias, the local variable alias that is introduced refers to a Module instance representing the module m3. However, because you mocked sys.modules['m2.m3'] beforehand, Python thinks that it has already discovered the module m2.m3, and returns the value of sys.modules['m2.m3']. So, the variable alias ends up referring to the Mock instance instead of a Module instance representing module m2.m3, and you get no errors, because the Mock instance accepts all calls.

The same thing happens when you use the import statement from m2 import m3; the variable m3 will end up referring to the Mock instance.

How To Fix Your Problem

As far as I know, you messed with Python's import system to a point where you cannot rely on it anymore to "just use" m2.m3 or m2.m3.m3. Python will find a way to complain in one way or another.

This is probably a situation where the actual problem is a design problem, and mocking is never going to be the right answer, and just cause more problems in the long run, however, I don't know what the actual situation is. However, you should try to find a way that avoids this whole situation.