python "No module named" error when calling a submodule

Question

I'm having trouble with the import statement as Python reports that there is no module named the_page or the_generator.

task_a.py contains import generator.the_page as ThePage, and when run as the main script, causes no problem.

the_generator.py contains import tasks.task_a, but when i run it as the main script, python throws the following error..

Traceback (most recent call last):
  File "/generator/the_generator.py", line 7, in <module>
    import tasks.tasks_a
  File "/generator/tasks/tasks_a.py", line 3, in <module>
    import generator.the_page as ThePage
ImportError: No module named the_page

Here's the structure.

generator/
    __init__.py
    the_generator.py
    the_page.py
    tasks/
        __init__.py
        task_a.py

Maybe you can help with my problem guys. Thanks for all the help!

Answer 1

Running scripts from the middle of a package is a bad idea, for a number of reasons, the most obvious of which is the one you're running into: When you import generator.the_generator somewhere, generator ends up as a package, so an absolute import of generator.the_page, or a relative import, will work fine. But when you just run the script generator/the_generator.py, there is no generator.the_generator, just __main__, and there is no generator package. The only other way Python could know how to find generator.the_page would be if the parent directory of generator were on sys.path, which it isn't.

As you can guess, you can work around this by munging sys.path to put the appropriate parent directory on there… but this is a bad idea too.

There are also many other problems with this solution. Most seriously, it can very easily lead to the same module being imported twice (because Python has no way of knowing that two apparently-unrelated names happen to refer to the same module). It's also hard to deploy (you can't install a script to /usr/local/bin if it depends on being inside a package…), it won't work if your package is run out of a .zip or .egg, etc.

---

There are two standard ways to solve this.

First, just run the script as a module rather than as a script. From the parent directory of generator, just python -m generator.the_generator instead of python generator/the_generator.py.

A major advantage of this is that it works just as well in normal installed deployments, when generator is in site-packages somewhere, as in testing.

Alternatively, create a script that sits alongside generator, and run that, not a module inside of it. This can be as trivial as moving all the if __name__ == '__main__': code in the_generator.py into a function, then writing two-line wrapper:

import generator.the_generator
generator.the_generator.main()

Again, this works just as well in normal installed deployments. Plus, it means the script can get installed into your bin directory, making things even easier, just like pip or ipython.