setuptools test hides import errors. How to have better info?

Question

In python setuptools, python setup.py test runs the testsuite. However if I have an import error in my testsuite, the only error message I obtain is an AttributeError complaining that my test class is missing. Is there a way to obtain a more detailed error message, so I can fix the testsuite ?

I will explain myself better with the following example. Suppose I have a package called foo, created anew with paster. I then add the test

./foo
./foo/__init__.py
./foo/tests
./foo/tests/__init__.py
./foo/tests/mytest.py
./setup.cfg
./setup.py

Now, suppose mytest.py contains the following code

import unittest
class MyTestClass(unittest.TestCase):
    def testFoo(self):
        self.assertEqual(1,1)

This works. However, if I try to import an unexistent module

import unittest
import frombiz
class MyTestClass(unittest.TestCase):
    def testFoo(self):
        self.assertEqual(1,1)

This is the error I obtain

Traceback (most recent call last):
  File "setup.py", line 26, in <module>
    test_suite = "foo.tests"
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/core.py", line 152, in setup
    dist.run_commands()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 975, in run_commands
    self.run_command(cmd)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/distutils/dist.py", line 995, in run_command
    cmd_obj.run()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/command/test.py", line 121, in run
    self.with_project_on_sys_path(self.run_tests)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/command/test.py", line 101, in with_project_on_sys_path
    func()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/command/test.py", line 130, in run_tests
    testLoader = loader_class()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 816, in __init__
    self.parseArgs(argv)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 843, in parseArgs
    self.createTests()
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 849, in createTests
    self.module)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 613, in loadTestsFromNames
    suites = [self.loadTestsFromName(name, module) for name in names]
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 587, in loadTestsFromName
    return self.loadTestsFromModule(obj)
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/Extras/lib/python/setuptools/command/test.py", line 34, in loadTestsFromModule
    tests.append(self.loadTestsFromName(submodule))
  File "/System/Library/Frameworks/Python.framework/Versions/2.6/lib/python2.6/unittest.py", line 584, in loadTestsFromName
    parent, obj = obj, getattr(obj, part)
AttributeError: 'module' object has no attribute 'mytest'

In other words, there's no reference to the failed import.

Answer 1

Use nose. No modification to your code should be necessary. Just do:

$ pip install nose
$ nosetests

If there is an ImportError, you will see it:

ERROR: Failure: ImportError (cannot import name MyModel)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "/home/garyvdm/bwreport/ve/lib/python3.2/site-packages/nose/failure.py", line 37, in runTest
    raise self.exc_class(self.exc_val).with_traceback(self.tb)
  File "/home/garyvdm/bwreport/ve/lib/python3.2/site-packages/nose/loader.py", line 390, in loadTestsFromName
    addr.filename, addr.module)
  File "/home/garyvdm/bwreport/ve/lib/python3.2/site-packages/nose/importer.py", line 39, in importFromPath
    return self.importFromDir(dir_path, fqname)
  File "/home/garyvdm/bwreport/ve/lib/python3.2/site-packages/nose/importer.py", line 86, in importFromDir
    mod = load_module(part_fqname, fh, filename, desc)
  File "/home/garyvdm/bwreport/bwreport/daemon/__init__.py", line 11, in <module>
    from bwreport.models import (
ImportError: cannot import name MyModel

Answer 2

The problem is that the first argument of __import__() must be module. When you need to reach some object with dotted name you never know what part is module and what isn't. The one way to get module.subname object is to try import it as submodule first and when it fails use getattr(module, subname), as unittest does. This will sometimes give you AttributeError instead of ImportError. Another way to do this is to try getattr(module, subname) first and only when it fails try import. This way isn't better: it will sometimes give you ImportError when more appropriate would be AttributeError.

Probably the best unittest can do in this case is to raise its own exception saying that both import and attribute lookup failed. Try to send bug report for this problem.