Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Best practice for lazy loading Python modules

Occasionally I want lazy module loading in Python. Usually because I want to keep runtime requirements or start-up times low and splitting the code into sub-modules would be cumbersome. A typical use case and my currently preferred implementation is this:

jinja2 = None

class Handler(...):
    ...
    def render_with_jinja2(self, values, template_name):
        global jinja2
        if not jinja2:
            import jinja2
        env = jinja2.Environment(...)
        ...

I wonder: is there a canonical/better way to implement lazy module loading?

like image 679
max Avatar asked Nov 14 '10 13:11

max


People also ask

How is lazy loading achieved when is it useful what are its pitfalls?

One form of lazy loading is infinity scroll, in which, the content of the web page is loaded as and when the user scrolls down the page. It is a popular technique being used by various websites. Advantages of Lazy loading: On-demand loading reduces time consumption and memory usage thereby optimizing content delivery.

Which of the following is the correct way to load module in Python?

Import in python is similar to #include header_file in C/C++. Python modules can get access to code from another module by importing the file/function using import. The import statement is the most common way of invoking the import machinery, but it is not the only way.

How do you import lazy in Python?

Lazy import is a very useful feature of the Pyforest library as this feature automatically imports the library for us, if we don't use the library it won't be added. This feature is very useful to those who don't want to write the import statements again and again in their code.


4 Answers

There's no reason for you to keep track of imports manually -- the VM maintains a list of modules that have already been imported, and any subsequent attempts to import that module result in a quick dict lookup in sys.modules and nothing else.

The difference between your code and

def render_with_jinja2(self, values, template_name):
    import jinja2
    env = jinja2.Environment(...)

is zero -- when we hit that code, if jinja2 hasn't been imported, it is imported then. If it already has been, execution continues on.

like image 81
bgporter Avatar answered Oct 05 '22 23:10

bgporter


class Handler(...):
    ...
    def render_with_jinja2(self, values, template_name):
        import jinja2
        env = jinja2.Environment(...)
        ...

There's no need to cache the imported module; Python does that already.

like image 33
Glenn Maynard Avatar answered Oct 06 '22 01:10

Glenn Maynard


The other answers have covered the actual details but if you are interested in a lazy loading library, check out apipkg which is part of the py package (py.test fame).

like image 30
Noufal Ibrahim Avatar answered Oct 06 '22 00:10

Noufal Ibrahim


Nice pattern from sqlalchemy: dependency injection:

@util.dependencies("sqlalchemy.orm.query")
def merge_result(query, *args):
    #...
    query.Query(...)

Instead of declaring all "import" statements at the top of the module, it will only import a module when it's actually needed by a function. This can resolve circular dependency problems.

like image 30
kolypto Avatar answered Oct 06 '22 01:10

kolypto