How to setup imports and dependencies between modules in the same python package

Question

I am writing a library in python. It has one package and 3 modules, such as

mypackage
    __init__.py
    utils.py
    fileio.py
    math.py

If I just leave the __init__.py empty, my users must figure out which functions are in which modules, which is annoying. The fact that I have three modules is an implementation detail that is not important.

Maybe I should import the main functions into the __init__.py, like this

from .utils import create_table
from .fileio import save_rows, load_rows
from .math import matrix_inverse

so that my users can just do

import mypackage as myp

rows = myp.load_rows()

Is that best practice?

What about the alternative to put ALL symbols into the __init__.py, such as

from .utils import *
from .fileio import *
from .math import *

And if there are any functions that I don't want to expose, I will prefix them with underscore. Is that better? It certainly is easier for me.

What if the fileio.py needs to call some functions in the utils.py? I could put

from .utils import *

into the fileio.py, but won't that create a circular or redundant reference? What's the best way to handle this?

Answer 1

Maybe I should import the main functions into the init.py, like this [...] Is that best practice?

I wouldn't say there is a "best practice", it depends on the specific case, but this is surely pretty common: you define a bunch of stuff in each module, and import the relevant ones in __init__. This is an easy way to not bother the users with remembering which submodule has the needed function, however it can get pretty annoying if you have a lot of functions to import from each module.

What about the alternative to put ALL symbols into the init.py, such as

from .utils import *
from .fileio import *
from .math import *

You most likely don't want to do this. This will import everything in user scripts, including other imported modules and internal functions. You should avoid it.

What if the fileio.py needs to call some functions in the utils.py? [...] won't that create a circular or redundant reference?

Yeah, that is something that can happen and you usually want to avoid it at all costs. If you need some functions from utils.py in fileio.py, you should import them explicitly as from .utils import x, y, z. Remember to also always use relative imports when importing things between modules of the same package (i.e. use from .utils import x, not from package.utils import x).

---

A good compromise between these two options you mention which solves most of the above problems (although not circular imports, you would have to avoid those yourself) would be to define an __all__ list inside each one of your modules to specify which functions should be exported when using from x import *, like this:

# utils.py

import sys

__all__ = ['user_api_one', 'user_api_two']

def user_api_one():
    ...

def user_api_two():
    ...

def internal_function():
    ...

If you properly define an __all__ list in all your modules, then in your __init__.py you will be able to safely do:

from .utils import *
from .fileio import *
from .math import *

This will only import relevant functions (for example user_api_one and user_api_two for utils, and not internal_function nor sys).