Mock an entire module in python

Question

I have an application that imports a module from PyPI. I want to write unittests for that application's source code, but I do not want to use the module from PyPI in those tests.
I want to mock it entirely (the testing machine will not contain that PyPI module, so any import will fail).

Currently, each time I try to load the class I want to test in the unittests, I immediately get an import error. so I thought about maybe using

try: 
    except ImportError:

and catch that import error, then use command_module.run(). This seems pretty risky/ugly and I was wondering if there's another way.

Another idea was writing an adapter to wrap that PyPI module, but I'm still working on that.

If you know any way I can mock an entire python package, I would appreciate it very much. Thanks.

Answer 1

While @Don Kirkby's answer is correct, you might want to look at the bigger picture. I borrowed the example from the accepted answer:

import pypilib

def bar(x):
    return pypilib.foo(x) + 1

Since pypilib is only available in production, it is not suprising that you have some trouble when you try to unit test bar. The function requires the external library to run, therefore it has to be tested with this library. What you need is an integration test.

That said, you might want to force unit testing, and that's generally a good idea because it will improve the confidence you (and others) have in the quality of your code. To widen the unit test area, you have to inject dependencies. Nothing prevents you (in Python!) from passing a module as a parameter (the type is types.ModuleType):

try:
    import pypilib     # production
except ImportError:
    pypilib = object() # testing

def bar(x, external_lib = pypilib):
    return external_lib.foo(x) + 1

Now, you can unit test the function:

import unittest
from unittest.mock import Mock

class Test(unittest.TestCase):
    def test_bar(self):
        external_lib = Mock(foo = lambda x: 3*x)
        self.assertEqual(10, bar(3, external_lib))
     

if __name__ == "__main__":
    unittest.main()

You might disapprove the design. The try/except part is a bit cumbersome, especially if you use the pypilib module in several modules of your application. And you have to add a parameter to each function that relies on the external library.

However, the idea to inject a dependency to the external library is useful, because you can control the input and test the output of your class methods, even if the external library is not within your control. Especially if the imported module is stateful, the state might be difficult to reproduce in a unit test. In this case, passing the module as a parameter may be a solution.

But the usual way to deal with this situation is called dependency inversion principle (the D of SOLID): you should define the (abstract) boundaries of your application, ie what you need from the outside world. Here, this is bar and other functions, preferably grouped in one or many classes:

import pypilib
import other_pypilib

class MyUtil:
    """
    All I need from outside world
    """
    @staticmethod
    def bar(x):
        return pypilib.foo(x) + 1

    @staticmethod
    def baz(x, y):
        return other_pypilib.foo(x, y) * 10.0

    ...
    # not every method has to be static

Each time you need one of these functions, just inject an instance of the class in your code:

class Application:
    def __init__(self, util: MyUtil):
        self._util = util
        
    def something(self, x, y):
        return self._util.baz(self._util.bar(x), y)

The MyUtil class must be as slim as possible, but must remain abstract from the underlying library. It is a tradeoff. Obviously, Application can be unit tested (just inject a Mock instead of an instance of MyUtil) while, under some circumstances (like a PyPi library not available during tests, a module that runs inside a framework only, etc.), MyUtil can be only tested within an integration test. If you need to unit test the boundaries of your application, you can use @Don Kirkby's method.

Note that the second benefit, after unit testing, is that if you change the libraries you are using (deprecation, license issue, cost, ...), you just have to rewrite the MyUtil class, using some other libraries or coding it from scratch. Your application is protected from the wild outside world.

Clean Code by Robert C. Martin has a full chapter on the boundaries.

Summary Before using @Don Kirkby's method or any other method, be sure to define the boundaries of your application irrespective of the specific libraries you are using. This, of course, does not apply to the Python standard library...

Answer 2

If you want to dig into the Python import system, I highly recommend David Beazley's talk.

As for your specific question, here is an example that tests a module when its dependency is missing.

bar.py - the module you want to test when my_bogus_module is missing

from my_bogus_module import foo

def bar(x):
    return foo(x) + 1

mock_bogus.py - a file in with your tests that will load a mock module

from mock import Mock
import sys
import types

module_name = 'my_bogus_module'
bogus_module = types.ModuleType(module_name)
sys.modules[module_name] = bogus_module
bogus_module.foo = Mock(name=module_name+'.foo')

test_bar.py - tests bar.py when my_bogus_module is not available

import unittest

from mock_bogus import bogus_module  # must import before bar module
from bar import bar

class TestBar(unittest.TestCase):
    def test_bar(self):
        bogus_module.foo.return_value = 99
        x = bar(42)

        self.assertEqual(100, x)

You should probably make that a little safer by checking that my_bogus_module isn't actually available when you run your test. You could also look at the pydoc.locate() method that will try to import something, and return None if it fails. It seems to be a public method, but it isn't really documented.