Python conditional 'module object has no attribute' error with personal package distinct from circular import issue

Question

I'm getting a 'module object has no attribute ..." error when trying to use a package heirarchy I created. The error is reminiscant of the error you get when there is a circular import (i.e. module a imports b and module b imports a), but I can't see that issue here. I've gone through many posts with a similar error, but none of the explanations I saw quite fit.

This was seen with python 2.7.1 and python 2.4.3.

I've watered it down to the following example:

Consider the following heirarchy (see code below):

alpha
alpha/__init__.py
alpha/bravo
alpha/bravo/__init__.py
alpha/bravo/charlie.py
alpha/bravo/delta.py
alpha/bravo/echo.py

The module charlie imports echo which in turn imports delta. If the alpha/bravo/__init__.py (like alpha/__init__.py) is essentially blank, a script can do:

import alpha.bravo.charlie

The problem surfaces if I try to import alpha.bravo.charlie in alpha/bravo/__init__.py (with the thinking I could surface relevant classes/methods there, and a script would do 'import alpha.bravo').

Code:

alpha/__init__.py

(blank)

alpha/bravo/__init__.py

import alpha.bravo.charlie

alpha/bravo/charlie.py

import alpha.bravo.echo
def charlie_foo(x): return str(x)
def charlie_bar(x): return alpha.bravo.echo.echo_biz()

alpha/bravo/delta.py

def delta_foo(x): return str(x)

alpha/bravo/echo.py

import alpha.bravo.delta
print alpha.bravo.delta.delta_foo(1)
def echo_biz(): return 'blah'

If I try:

python -c 'import alpha.bravo'

I get:

Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/kmkc980/svn/working/azcif/python/lib/alpha/bravo/__init__.py", line 1, in <module>
    import alpha.bravo.charlie
  File "/home/kmkc980/svn/working/azcif/python/lib/alpha/bravo/charlie.py", line 1, in <module>
    import alpha.bravo.echo
  File "/home/kmkc980/svn/working/azcif/python/lib/alpha/bravo/echo.py", line 2, in <module>
    print alpha.bravo.delta.delta_foo(1)
AttributeError: 'module' object has no attribute 'bravo'

But, if I comment out the import line in alpha/bravo/__init__.py, then all seems OK:

python -c 'import alpha.bravo'

python -c 'import alpha.bravo.charlie'
1

Moreover, if I use the same code above (including the import line in alpha/bravo/__init__.py), but edit everything to exclude the 'alpha' level of the hierarchy, it seems to work fine.

So the hierarchy is now just:

bravo
bravo/__init__.py
bravo/charlie.py
bravo/delta.py
bravo/echo.py

and I change all the lines with "alpha.bravo.*" to "bravo.*"

Then no problem:

python -c 'import bravo'
1

I've been able to work around the issue, but I'd still like to understand it. Thanks.

Answer 1

Here's the why

(This is, I believe, mostly supported by the explanation at http://docs.python.org/faq/programming.html#how-can-i-have-modules-that-mutually-import-each-other)

When the Python interpreter encounters a line of the form import a.b.c, it runs through the following steps. In pseudo-python:

for module in ['a', 'a.b', 'a.b.c']:
    if module not in sys.modules:
        sys.modules[module] = (A new empty module object)
        run every line of code in module # this may recursively call import
        add the module to its parent's namespace
return module 'a'

There are three important points here:

1. The modules a, a.b, and a.b.c get imported in order, if they haven't been imported already

2. A module does not exist in its parent's namespace until it has completely finished being imported. So module a does not have a b attribute until a.b has been imported completely.

3. No matter how deep your module chain is, even if you import a.b.c.d.e.f.g, your code only gets one symbol added to its namespace: a.

So when you later try to run a.b.c.d.e.f.g.some_function(), the interpreter has to traverse all the way down the chain of modules to get to that method.

Here's what is happening

Based on the code that you have posted, the problem seems to lie in the print statement in alpha/bravo/echo/__init__.py. What the interpreter has done by the time it gets there is roughly this:

1. Set up an empty module object for alpha in sys.modules

2. Run the code in alpha/__init__.py (Note that dir(alpha) won't contain 'bravo' at this point)

3. Set up an empty module object for alpha.bravo in sys.modules

4. Run the code in alpha/bravo/__init__.py:

4.1 Set up an empty module object for alpha.bravo.charlie in sys.modules

4.2 Run the code in alpha/bravo/charlie/__init__.py:

4.2.1 Set up an empty module object for alpha/bravo/echo in sys.modules

4.2.2 Run the code in alpha/bravo/echo/__init__.py:

4.2.2.1 Set up an empty module object for alpha/bravo/delta in sys.modules

4.2.2.2 Run the code in alpha/bravo/delta/__init__.py -- This finishes, so 'delta' is added to 'alpha.bravo's symbols.

4.2.2.3 Add 'alpha' to echo's symbols. This is the last step in import alpha.bravo.delta.

At this point, if we call dir() on all of the modules in sys.modules, we will see this:

• 'alpha': ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__return__'] (this is essentially empty)

• 'alpha.bravo': ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', 'delta'] (delta has finished being imported, so it's here)

• 'alpha.bravo.charlie': ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__'] (empty)

• 'alpha.bravo.delta': ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__return__', 'delta.foo'] (This is the only one that has completed)

• 'alpha.bravo.echo': ['__builtins__', '__doc__', '__file__', '__name__', '__package__', '__path__', '__return__', 'alpha']

Now the intepreter continues with alpha/bravo/echo/__init__.py, where it encounters the line print alpha.bravo.delta.delta_foo(1). That starts this sequence:

1. get the global variable alpha -- this returns the still-empty alpha module.
2. call getattr(alpha, 'bravo') -- this fails, because alpha.bravo isn't finished being initialized yet, so bravo hasn't been inserted into alpha's symbol table.

This is the same thing that happens during a circular import -- the module isn't finished being initialized, so the symbol table isn't completely updated, and attribute access fails.

If you were to replace the offending line in echo/__init__.py with this:

import sys
sys.modules['alpha.bravo.delta'].delta_foo(1)

That would probably work, since delta is completely initialized. But until bravo is complete (after echo and charlie return), the symbol table for alpha won't be updated, and you won't be able to access bravo through it.

Also, as @Rik Poggi says, if you change the import line to

from alpha.bravo.delta import delta_foo

Then that will work. In this case, because from alpha.bravo.delta goes right to the sys.modules dict, rather than traversing from alpha to bravo to delta, it can get the function from the delta module and assign it to a local variable, which you can then access without any trouble.

Answer 2

Instead of using absolute imports, it might help to use relatives.

i.e.

alpha/bravo/_init_.py

import alpha.bravo.charlie

should be

import charlie

Otherwise, it probably is a circular import. i.e. if you import alpha.bravo.charlie from charlie, that means

alpha/__init__.py
bravo/__init__.py
charlie/__init__.py 

All are loaded (or rather, prevented from doing so since they're already loaded). That might cause the problem you're seeing.