Avoiding pylint complaints when importing Python packages from submodules

Question

Background

I have a Python application dependent upon another package which is provided as a git submodule, yielding a directory structure similar to the following:

foo/
    bar/
        bar/
            __init__.py
            eggs.py
        test/
        setup.py
    foo/
        __init__.py
        ham.py
    main.py

Accessing the foo package is simple enough, as main.py is executed from the toplevel foo/ directory; but the bar package is nested within another bar directory and is not directly importable.

This is solvable readily enough, by modifying sys.path at the beginning of main.py:

import sys

# Or sys.path.append()
sys.path.insert(0, './bar')

from bar.eggs import Eggs
from foo.ham import Ham

(Note: this code example assumes that main.py will always be invoked from foo/; in cases where this may not be the case, '.bar' could be replaced with os.path.join(os.path.dirname(__file__), 'bar') though this is clearly more unwieldy.)

The Problem

Unfortunately, pylint doesn't like this solution. While the code works, the linter considers the sys.path modifications to be a block of code ending the "top of the module" and gives an undesirable wrong-import-position warning:

C: 6, 0: Import "from bar.eggs import Eggs" should be placed at the top of the module (wrong-import-position)
C: 7, 0: Import "from foo.ham import Ham" should be placed at the top of the module (wrong-import-position)

---

Similar questions

Adding a path to sys.path in python and pylint

This questioner has an issue with pylint failing to correctly parse the imports altogether. The lone answer to the this question suggests adding to pylint's internal path; this does nothing to avoid complaints about an interleaved sys.path modification.

Answer 1

Configure pylint

Disabling the wrong-import-position checker in .pylintrc is the simplest solution, but throws away valid warnings.

A better solution is to tell pylint to ignore the wrong-import-position for these imports, inline. The false-positive imports can be nested in an enable-disable block without losing any coverage elsewhere:

import sys

sys.path.insert(0, './bar')

#pylint: disable=wrong-import-position

from bar.eggs import Eggs
from foo.ham import Ham

#pylint: enable=wrong-import-position

Ham()

# Still caught
import something_else

However, this does have the slight downside of funkiness if wrong-import-order is ever disabled in .pylintrc.

---

Avoid modifying sys.path

Sometimes unwanted linting warnings stem from going about a problem incorrectly to start with. I've come up with a number of ways to avoid modifying sys.path in the first place, though they are not applicable to my own situation.

Perhaps the most straightforward method is to modify PYTHONPATH to include the submodule directory. This, however, must then either be specified each time the application is invoked, or modified on a system/user level, potentially harming other processes. The variable could be set in a wrapping shell or batch script, but this requires either further environmental assumptions or limits changes to the invocation of Python.

A more modern and less trouble-fraught analog is to install the application in a virtual environment and simply add the submodule path to the virtual environment.

Reaching further afield, if the submodule includes a setuptools setup.py, it may simply be installed, avoiding path customization altogether. This may be accomplished by maintaining a publication to repositories such as pypi (a non-starter for proprietary packages) or by utilizing/abusing pip install -e to install either the submodule package directly or from its repository. Once again, virtual environments make this solution simpler by avoiding potential cross-application conflicts and permissions issues.

If the target OS set can be limited to those with strong symlink support (in practice this excludes all Windows through at least 10), the submodules may be linked into to bypass the wrapping directory and put the target package directly in the working directory:

foo/
    bar/ --> bar_src/bar
    bar_src/
        bar/
            __init__.py
            eggs.py
        test/
        setup.py
    foo/
        __init__.py
        ham.py
    main.py

This has the downside of limiting the potential users of the application and filling the foo directory with confusing clutter, but may be an acceptable solution in some cases.

Answer 2

Hard Coded Locations

The problem with this set up is that it makes very specific assumptions about the locations of files. In particular, it hard codes a location to another package.

In your case, you hard code it to a relative path. This additionally imposes a requirement on the end user to have a very specific current directory. This is annoying if you're an end user. If I have a file I want to use as input to your code, I should be able to have my current directory be my user home path (~ in Linux, %USERPROFILE% in Windows) and pass in a relative path to my file while using an absolute path to the script itself. (E.g., python /path/to/your/script ./myinput.txt.) Hard coding locations like this makes it impossible to do. I also note that your bar directory contains a setup.py, implying it's a standalone package. Wonderful. What if I want to run main.py again a specific version of the package installed somewhere? Again, this is not possible with the modification to sys.path your script performs.

The only locations you should ever hard code in code are locations to resources that will be distributed directly with the code, always in the same location, like if you had a recipes.dat file alongside eggs.py. In such cases, the paths should be relative to the script's (or the binary's in other languages) current location. (E.g., RECIPES_PATH = os.path.join(os.path.dirname(__name__), 'recipes.dat').) When you have a separate package, it might be somewhere else than where your main.py script expects.

Let Python Do Its Job

Finding and loading packages is a fundamental feature to Python. Let it do that. And when you run into a situation where it can't find it out of the box (because your code is not installed anywhere), use the standard mechanisms to work with them.

The PYTHONPATH environment variable is probably the simplest way of dealing with it. And it's easy. You just need a companion script to set up the command line environment:

setupenv.sh:

DIR="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )" # See https://stackoverflow.com/a/246128/1394393

if [ -n "$PYTHONPATH" ]; then
    PYTHONPATH=$PYTHONPATH:
fi
PYTHONPATH=$PYTHONPATH${DIR%%/}/bar

Then:

$ source setupenv.sh
$ python ./main.py

(It's equally simple to do this in a Windows batch/cmd file as well.)

Okay, it's a little annoying to have to set up the environment each time you start up your terminal when you're actively developing the code. But it's not that bad. I do this on my own projects, and it's something I do in the morning and don't think about again until I launch a new terminal. (My script sets up more besides: activates a virtual environment, sets PATH for some native binaries.) And it's much, much cleaner for the project.

You might argue: "Well, we're still hard coding the location in the sh file." Yes, we are. But this script is part of the repository. Note that the path I use is relative to the script itself; that's because I know how the code repository is structured. I don't know the user's current directory when they're working at the command line, and I certainly don't know where main.py is going to be distributed. Perhaps it'll end up in its own package in the final destination. Regardless, it's not that script's job to know where other packages lives. That is the job of this setupenv.sh script, within this repository.