How can I lazily import a module in python?

Question

I have classes which require dependencies in order to be instantiated but are otherwise optional. I'd like to lazily import the dependencies and fail to instantiate the class if they aren't available. Note that these dependencies are not required at the package level (otherwise they'd be mandatory via setuptools). I currently have something like this:

class Foo:
    def __init__(self):
        try:
            import module
        except ImportError:
            raise ModuleNotFoundError("...")

    def foo(self):
        import module

Because this try/except pattern is common, I'd like to abstract it into a lazy importer. Ideally if module is available, I won't need to import it again in Foo.foo so I'd like module to be available once it's been imported in __init__. I've tried the following, which populates globals() and fails to instantiate the class if numpy isn't available, but it pollutes the global namespace.

def lazy_import(name, as_=None):
    # Doesn't handle error_msg well yet
    import importlib
    mod = importlib.import_module(name)
    if as_ is not None:
        name = as_
    # yuck...
    globals()[name] = mod

class NeedsNumpyFoo:
    def __init__(self):
        lazy_import("numpy", as_="np")

    def foo(self):
        return np.array([1,2,])

I could instantiate the module outside the class and point to the imported module if import doesn't fail, but that is the same as the globals() approach. Alternatively lazy_import could return the mod and I could call it whenever the module is needed, but this is tantamount to just importing it everywhere as before.

Is there a better way to handle this?

Answer 1

Pandas actually has a function import_optional_dependency which may make a good example (link GitHub) as used in SQLAlchemyEngine (link GitHub)

However, this is only used during class __init__ to get a meaningful error (raise ImportError(...) by default!) or warn about absence or old dependencies (which is likely a more practical use of it, as older or newer dependencies may import correctly anywhere if they exist, but not work or be explicitly tested against or even be an accidental local import)

I'd consider doing similarly, and either not bother to have special handling or only do it in the __init__ (and then perhaps only for a few cases where you're interested in the version, etc.) and otherwise simply import where needed

class Foo():
    def __init__(self, ...):
        import bar  # only tests for existence

    def usebar(self, value):
        import bar
        bar.baz(value)

---

Plausibly you could assign to a property of the class, but this may cause some trouble or confusion (as the import should already be available in globals once imported)

class Foo():
    def __init__(self, ...):
        import bar
        self.bar = bar

    def usebar(self, value):
        self.bar.baz(value)

Answer 2

You're overthinking this. A better solution is a general lazy loader with these properties:

• drop-in replacement for import statement. In other words: can be used for global or local imports, no extra checks at point of use, same syntax as regular module usage
• no messing with globals() dict
• can be used to import 3rd party libs, i.e. no code changes
• minimal performance overhead

The fact that you load the module in a class shouldn't be part of the solution. It overcomplicates the problem.

Solution

Here's my solution with those properties:

class LazyLoader () :
    'thin shell class to wrap modules.  load real module on first access and pass thru'

    def __init__ (me, modname) :
        me._modname  = modname
        me._mod      = none
   
    def __getattr__ (me, attr) :
        'import module on first attribute access'

        try :
            return getattr (me._mod, attr)
        
        except Exception as e :
            if me._mod is none :
                # module is unset, load it
                me._mod = importlib.import_module (me._modname)
            else :
                # module is set, got different exception from getattr ().  reraise it
                raise e

        # retry getattr if module was just loaded for first time
        # call this outside exception handler in case it raises new exception
        return getattr (me._mod, attr)

# end class            

Usage

import sys
math = LazyLoader ('math')  # equivalent to : import math

math.ceil (1.7)  # module loaded here on first access

# OP's use case
bar = LazyLoader ('bar')

class Foo():
    def __init__(self, ...):
        pass

    def usebar(self, value):
        bar.baz(value)

# this will raise an exception in usebar when bar isn't available, same as OP's code
# you can catch and handle this inside usebar, or just let it propagate
# if you prefer to detect whether bar is available inside init, just do this: 

def __init__(self, ...):
    try :
        bar.baz
    except :
        # handle when bar isnt present

Performance

# Regular top-level import
> python3 -m timeit -s 'import math' -c 'math.floor'
10000000 loops, best of 5: 32.4 nsec per loop

# Local import (same with or without -s)
> python3 -m timeit -s 'import math' -c 'import math ; math.floor'
2000000 loops, best of 5: 126 nsec per loop

# LazyLoader with exceptions
> python3 -m timeit -s 'import LazyLoader ; math = LazyLoader ("math")' -c 'math.floor'
500000 loops, best of 5: 453 nsec per loop

# LazyLoader with "if mod == none"
> python3 -m timeit -s 'import LazyLoader ; math = LazyLoader ("math")' -c 'math.floor'
500000 loops, best of 5: 540 nsec per loop

# for comparison :
> python3 -m timeit -s 'import os; path = "/foo/bar"' -c 'os.path.split (path)'
500000 loops, best of 5: 879 nsec per loop

A 400 ns penalty is quite good. 1000 calls adds 400 microsec. A million calls adds 400 millisec. Unless your called function is very fast and you're making an ungodly number of calls, you shouldn't notice any difference.