Optional, dynamic dependencies at Python runtime

Question

I'm building a complex Python application with a maximalist approach in terms of support for different libraries, some of which are machine learning libraries that use several gigabytes. I am now getting ready to make a release of the project, but I've realized the build size is enormous and 99% of it is not my code, and some of it may never be called by the user.

What I'd like to be able to do is have the user download a release with the minimal dependencies, and if they choose to use other ones in the GUI, prompt them and install them at runtime, and maybe even rebuild the application with pyinstaller.

I don't expect all my users to be developers, so just having a configurable build script isn't the best option, and I don't want to ship multiple releases.

Is what I'm describing possible with Python, and is it a common pattern? How would I accomplish something like this?

If this question is better suited to the Software Engineering stack exchange, I can move it there.

EDIT: Some of my packages are also Linux-only or Windows-only. I want to release my project as a portable executable with something like PyInstaller. I don't expect my users to understand how to manage a venv or system Python version either. I suppose I could understand making another piece of software that installs and builds the program by and manages its own Python version, venv, and executable like an installed system package. Is such a complicated, overbearing solution necessary to handle this?

Answer 1

It is possible with Python what what you're describing, and it's actually a common pattern in software development, known as lazy loading or dynamic dependencies. It allows you to defer the installation and loading of certain dependencies until they are actually needed by the user.

• you can use Python's built-in importlib module, which provides functions for programmatically importing modules at runtime.
• Identify the optional dependencies that you want to include in your application but not ship by default. libraries that are not essential for the core functionality of your application.
• Modify your code to use dynamic imports for these optional dependencies like:

import importlib

def use_optional_library():
    try:
        optional_lib = importlib.import_module('optional_lib')
        # Use the optional library here
    except ImportError:
        # Handle the case when the library is not available
        pass

• In your GUI, provide a way for the user to enable or disable the usage of these optional libraries. When the user chooses to enable a specific library, you can call the corresponding function that uses importlib.import_module() to load the library dynamically.
• To handle the installation of the optional dependencies, you can use a package manager like pip or conda or library like pipenv or poetry.
• You can use the subprocess module to execute the package manager commands programmatically.
• distribute your application as a portable executable using PyInstaller, you can include the optional libraries as data files rather than as direct dependencies. This way, they won't be bundled with the executable by default, but you can load them dynamically at runtime if the user chooses to enable them.

This approach adds some complexity to your codebase, it allows you to provide a more flexible and customizable experience for your users without bloating the initial download size.