Python Imports, Paths, Directories & Modules

Question

Let me start by saying I've done extensive research over the course of the past week and have not yet found actual answers to these questions - just some fuzzy answers that don't really explain what is going on. If that's just cause I missed what I was looking for, I'm sorry - please just point me in the correct direction.

My directory structure is:

TestProject/
    runtest*
    testpackage/
        __init__.py
        testmod.py
        testmod2.py
        testsubs/
            testsubmod.py

A couple notes:

• I'm using python2.7 on Ubuntu
• I'm testing with bpython
• I'm running bpython from specific directories to test the way import behaves
• I'm attempting to follow best practices.
• This package is not installed, it's in a random dev directory
• This directory is not in the pythonpath
• I have a single init.py in the package directory
• There are no init.py files in the nested directories
• The init.py file is empty
• testpackage/testmod.py contains TestModClass
• testpackage/testsubs/testsubmod.py contains TestSubModClass

Things I've observed:

• When I run bpython from TestProject/ import testpackage works
  • This does not import testpackage.testmod
  • I cannot access testpackage.testmod at all
• When I run bpython from TestProject/ import testpackage.testmod fails
• When I run bpython from TestProject/ from testpackage import testmod works
• I can add code to init.py to explicitly import testmod.py, but not testsubs/testmod.py
  • I don't think this is the correct way to do this, what if the user doesn't want to import that module?
• from testmod.py i can import testmod2, but not testpackage.testmod2
  • This would be nice to do so that I can name my own modules with overlapping names with STL or twisted (such as testpackage.logging) without causing errors (it sucks to have to name my own modules crap like customerlogging, instead of just mypackage.logging)

And the questions:

1. Does python deal differently with imports on packages & modules that exist in the pythonpath than when you are trying to import from your current directory?
2. Why doesn't import testpackage give me access to testpackage.testmod? When I import os, I can then access os.path (etc).
3. With a package, should I stick to using a single init.py in the base directory, or should I be nesting them in subsequent directories?
4. How can I import a module specifying the package name? I.E. from testmod.py, I would like to import testpackage.testmod2 rather than just testmod2.
5. What is the proper way to import submodules from the subsubs directory?
   1. The only solution I see is to append that directory to the pythonpath from init.py, but I don't know if that's the correct way.

Thanks in advance.

Answer 1

First off, you will find all the information you need in section 6 of The Python Tutorial.

---

(1) Does python deal differently with imports on packages & modules that exist in the pythonpath than when you are trying to import from your current directory?

No, it doesn't. Actually, Python always searches sys.path when importing modules. Modules in the current directory are only found since sys.path contains an entry with the empty string, meaning the current directory.

---

(2) Why doesn't import testpackage give me access to testpackage.testmod? When I import os, I can then access os.path (etc).

For efficiency, import testpackage only loads testpackage/__init__.py. If you need testpackage.testmod, you have to explicitely import it:

import testpackage   # Just imports testpackage, not testpackage.testmod!
import testpackage.testmod   # Import *both* testpackage and testpackage.testmod!

If you always want to export testmod, import it within __init__.py, this is what os (os/__init__.py) does. This way, testpackage.testmod is always available implicitely if you import testpackage.

Since Python is cross-platform, there is actually no way to consistently and automatically load modules in a directory, because some filesystems are case-insensitive (Windows!). Python would not know whether to load os/path.py as os.path or os.Path, etc.

---

(3) With a package, should I stick to using a single __init__.py in the base directory, or should I be nesting them in subsequent directories?

You always need an __init__.py for each subpackage. There were discussions on dropping this requirement, but it was decided to keep it as it is.

---

(4) How can I import a module specifying the package name? I.E. from testmod.py, I would like to import testpackage.testmod2 rather than just testmod2.

This should work. Just ensure that you run the code from the top-level directory. If the current directory is testpackage, testmod does not know that it's in a package.

The preferred way is to use relative intra-package imports, though:

from . import testmod2

This prevents name clashes if there is a global module named testmod2 and enables you to use the names of well-known modules within your package without problems.

---

(5) What is the proper way to import submodules from the subsubs directory? The only solution I see is to append that directory to the pythonpath from __init__.py, but I don't know if that's the correct way.

No, don't do that! Never, ever put a directory to sys.path when one of it's parent directories already is in sys.path! This could cause your modules to be loaded twice, which is a bad thing!

Usually you should be able to load modules from subpackages using absolute or relative imports:

import testpackage.testsubs.testsubmod
from testpackage.testsubs import testsubmod
from .testsubs import testsubmod

Just make sure to create a __init__.py within testsubs/!