Python Not Finding Module

Question

Given the following python project, created in PyDev:

├── algorithms
│   ├── __init__.py
│   └── neighborhood
│       ├── __init__.py
│       ├── neighbor
│       │   ├── connector.py
│       │   ├── __init__.py
│       │   ├── manager.py
│       │   └── references.py
│       ├── neighborhood.py
│       ├── tests
│       │   ├── fixtures
│       │   │   └── neighborhood
│       │   ├── __init__.py
│       └── web
│           ├── __init__.py
│           └── service.py
├── configuration
│   ├── Config.py
│   └── __init__.py
├── __init__.py
└── webtrack
    |- teste.py
    ├── .gitignore
    ├── __init__.py
    ├── manager
        ├── Data.py
        ├── ImportFile.py
        └── __init__.py

We've been trying with no success to import modules from one folder to another, such as:

from algorithms.neighborhood.neighbor.connector import NeighborhoodConnector

Which yields the result:

Traceback (most recent call last):
File "teste.py", line 49, in <module>
from algorithms.neighborhood.neighbor.connector import NeighborhoodConnector
ImportError: No module named algorithms.neighborhood.neighbor.connector

We tried to append its path to the sys.path variable but with no success.

We also tried to use os.walk to insert all paths into PATH variable but still we get the same error, even though we checked PATH does contain the path to find the modules.

We are using Python 2.7 on Linux Ubuntu 13.10.

Is there anything we could be doing wrong?

Thanks in advance,

Answer 1

Getting imports right when running a script that lives within a package is tricky. You can read this section of the (sadly deferred) PEP 395 for a description of a bunch of ways that don't work to run such a script.

Give a file system hierarchy like:

top_level/
    my_package/
        __init__.py
        sub_package/
            __init__.py
            module_a.py
            module_b.py
            sub_sub_package/
                __init__.py
                module_c.py
        scripts/
            __init__.py
            my_script.py
            script_subpackage/
                 __init__.py
                 script_module.py

There are only a few ways to make running my_script.py work right.

1. The first would be to put the top_level folder into the PYTHONPATH environment variable, or use a .pth file to achieve the same thing. Or, once the interpreter is running, insert that folder into sys.path (but this can get ugly).

   Note that you're adding top_level to the path, not my_package! I suspect this is what you've got messed up in your current attempts at this solution. Its very easy to get wrong.

   Then, absolute imports like import my_package.sub_package.module_a will mostly work correctly. (Just don't try importing package.scripts.my_script itself while it is running as the __main__ module, or you'll get a weird duplicate copy of the module.)

   However, absolute imports will always be more verbose than relative imports, since you always need to specify the full path, even if you're importing a sibling module (or "niece" module, like module_c from module_a). With absolute imports, the way to get module_c is always the big, ugly mouthful of code from my_package.sub_package.sub_sub_package import module_c regardless of what module is doing the importing.

2. For that reason, using relative imports is often more elegant. Alas, they're hard to get to work from a script. The only ways are:

   1. Run my_script from the top_level folder with the -m flag (e.g. python -m my_package.scripts.my_script) and never by filename.

      It won't work if you're in a different folder, or if you use a different method to run the script (like pressing F5 in an IDE). This is somewhat inflexible, but there's not really any way to make it easier (until PEP 395 gets undeferred and implemented).

   2. Set up sys.path like for absolute imports (e.g. add top_level to PYTHONPATH or something), then use a PEP 366 __package__ string to tell Python what the expected package of your script is. That is, in my_script.py you'd want to put something like this above all your relative imports:

      if __name__ == "__main__" and __package__ is None:
          __package__ = "my_package.my_scripts"
      

      This will require updating if you reorganize your file organization and move the script to a different package (but that's probably less work than updating lots of absolute imports).

   Once you've implemented one of those soutions, your imports can get simpler. Importing module_c from module_a becomes from .sub_sub_package import module_c. In my_script, relative imports like from ..subpackage import module_a will just work.