Why does import error change to "cannot import name" on the second import?

Question

Here's a mysterious python problem:

I'm developing a python package that occasionally reports import errors looking like ImportError: cannot import name …. The modules it cannot import generally

• are importable
• do not have any circular import issues (that I can detect).

I have been able to reproduce a similar effect with this simple example:

mypkg/__init__.py:

    from . import module_a
    yarg  ## cause import error

mypkg/module_a.py:

    print "imported module_a"

Now I will attempt to import the package twice. Notice that the error changes on the second import:

>>> import mypkg
Module A imported
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mypkg/__init__.py", line 2, in <module>
    yarg
NameError: name 'yarg' is not defined
>>> import mypkg
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mypkg/__init__.py", line 1, in <module>
    from . import module_a
ImportError: cannot import name module_a

What gives?

Note:

• the problem goes away if I use an absolute import instead
• if I delete the key sys.modules['mypkg.module_a'] after the first import, then the second import gives me back the original error message

Answer 1

I can illustrate what is causing the difference between each import, but I'm not expert enough on Python's import process to be able to explain the why very well.

>>> import sys
>>> before_import = set(sys.modules.keys())
>>> import mypkg
imported module_a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mypkg\__init__.py", line 2, in <module>
    yarg  ## cause import error
NameError: name 'yarg' is not defined
>>> after_import = set(sys.modules.keys())
>>> after_import.difference(before_import)
set(['mypkg.module_a'])

When you import mypkg, it successfully imports module_a and adds it to sys.modules. Then mypkg errors and doesn't get added itself to the sys.modules dictionary. Deleting the entry allows you to reimport with the same error:

>>> import sys
>>> del sys.modules['mypkg.module_a']
>>> import mypkg
imported module_a
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "mypkg\__init__.py", line 2, in <module>
    yarg  ## cause import error
NameError: name 'yarg' is not defined

Now, what I think is happening is:

1. import mypkg starts the import process for mypkg

2. As it's processing mypkg, it successfully imports module_a as a subpackage of itself and adds it to sys.modules

3. When it hits the error, the import process for mypkg fails and no entry for mypkg is left in sys.modules

4. The conjunction of the package failing but the subpackage succeeding conflicts with subsequent imports

That's about the best I can fathom, sorry. Python's import process is something of a black art.

Answer 2

I'm pretty sure the problem is that your package is failing to load. You've put some nonsense (yarg by itself) in the __init__.py file. This means that mypkg can't be imported. Because of this, mypkg.module_a can't be imported either.

I suspect you get different errors because Python is doing some caching of the module state. The first time you try importing mypkg the import of its submodule module_a is allowed even though mypkg is in the process of being loaded. The second time, the fact that mypkg doesn't work right is cached, so mypkg.module_a fails to load since its parent package is broken.