Good way to collect programmatically generated test suites in nose or pytest

Question

Say I've got a test suite like this:

class SafeTests(unittest.TestCase):
    # snip 20 test functions

class BombTests(unittest.TestCase):
    # snip 10 different test cases

I am currently doing the following:

suite = unittest.TestSuite()
loader = unittest.TestLoader()
safetests = loader.loadTestsFromTestCase(SafeTests)
suite.addTests(safetests)

if TARGET != 'prod':
    unsafetests = loader.loadTestsFromTestCase(BombTests)
    suite.addTests(unsafetests)


unittest.TextTestRunner().run(suite)

I have major problem, and one interesting point

• I would like to be using nose or py.test (doestn't really matter which)

• I have a large number of different applications that are exposing these test suites via entry points.

  I would like to be able to aggregate these custom tests across all installed applications so I can't just use a clever naming convention. I don't particularly care about these being exposed through entry points, but I do care about being able to run tests across applications in site-packages. (Without just importing... every module.)

I do not care about maintaining the current dependency on unittest.TestCase, trashing that dependency is practically a goal.

---

EDIT This is to confirm that @Oleksiy's point about passing args to nose.run does in fact work with some caveats.

Things that do not work:

• passing all the files that one wants to execute (which, weird)
• passing all the modules that one wants to execute. (This either executes nothing, the wrong thing, or too many things. Interesting case of 0, 1 or many, perhaps?)
• Passing in the modules before the directories: the directories have to come first, or else you will get duplicate tests.

This fragility is absurd, if you've got ideas for improving it I welcome comments, or I set up a github repo with my experiments trying to get this to work.

All that aside, The following works, including picking up multiple projects installed into site-packages:

#!python
import importlib, os, sys
import nose

def runtests():
    modnames = []
    dirs = set()
    for modname in sys.argv[1:]:
        modnames.append(modname)

        mod = importlib.import_module(modname)
        fname = mod.__file__
        dirs.add(os.path.dirname(fname))

    modnames = list(dirs) + modnames

    nose.run(argv=modnames)

if __name__ == '__main__':
    runtests()

which, if saved into a runtests.py file, does the right thing when run as:

runtests.py project.tests otherproject.tests

Answer 1

For nose you can have both tests in place and select which one to run using attribute plugin, which is great for selecting which tests to run. I would keep both tests and assign attributes to them:

from nose.plugins.attrib import attr

@attr("safe")
class SafeTests(unittest.TestCase):
    # snip 20 test functions

class BombTests(unittest.TestCase):
    # snip 10 different test cases

For you production code I would just call nose with nosetests -a safe, or setting NOSE_ATTR=safe in your os production test environment, or call run method on nose object to run it natively in python with -a command line options based on your TARGET:

import sys
import nose

if __name__ == '__main__':
    module_name = sys.modules[__name__].__file__
    argv = [sys.argv[0], module_name]
    if TARGET == 'prod':
        argv.append('-a slow')

    result = nose.run(argv=argv)

Finally, if for some reason your tests are not discovered you can explicitly mark them as test with @istest attribute (from nose.tools import istest)

Answer 2

This turned out to be a mess: Nose pretty much exclusively uses the TestLoader.load_tests_from_names function (it's the only function tested in unit_tests/test_loader) so since I wanted to actually load things from an arbitrary python object I seemed to need to write my own figure out what kind of load function to use.

Then, in addition, to correctly get things to work like the nosetests script I needed to import a large number of things. I'm not at all certain that this is the best way to do things, not even kind of. But this is a stripped down example (no error checking, less verbosity) that is working for me:

import sys
import types
import unittest

from nose.config import Config, all_config_files
from nose.core import run
from nose.loader import TestLoader
from nose.suite import ContextSuite
from nose.plugins.manager import PluginManager

from myapp import find_test_objects

def load_tests(config, obj):
    """Load tests from an object

    Requires an already configured nose.config.Config object.

    Returns a nose.suite.ContextSuite so that nose can actually give
    formatted output.
    """

    loader = TestLoader()
    kinds = [
        (unittest.TestCase, loader.loadTestsFromTestCase),
        (types.ModuleType, loader.loadTestsFromModule),
        (object, loader.loadTestsFromTestClass),
    ]
    tests = None
    for kind, load in kinds.items():
        if isinstance(obj, kind) or issubclass(obj, kind):
            log.debug("found tests for %s as %s", obj, kind)
            tests = load(obj)
            break

    suite = ContextSuite(tests=tests, context=obj, config=config)

def main():
    "Actually configure the nose config object and run the tests"
    config = Config(files=all_config_files(), plugins=PluginManager())
    config.configure(argv=sys.argv)

    tests = []
    for group in find_test_objects():
        tests.append(load_tests(config, group))

    run(suite=tests)