Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Running subset of auto-discovered python unittests

Short Question
Is it possible to select, at run time, what unittests are going to be run when using an auto-discovery method in Python's unittest module.

Background
I am using the unittest module to run system tests on an external system. See below for an example sudo-testcase. The unittest module allows me to create an arbitrary number testcases that I can run using the unittest's testrunner. I have been using this method roughly 6 months of constant use and it is working out very well.

At this point in time I am wanting to try and make this more generic and user friendly. For all of my test suites that I am running now, I have hard coded which tests must run for every system. This is fine for an untested system, but when a test fails incorrectly (a user connects to the wrong test point etc...) they must re-run the entire test suite. As some of the complete suites can take up to 20 min, this is no where near ideal.

I know it is possible to create custom testsuite builders that could define which tests to run. My issue with this is that there are hundreds of testcases that can be run and maintaining this would be come a nightmare if test case names change etc...

My hope was to use nose, or the built-in unittest module to achieve this. The discovery part seems to be pretty straight forward for both options, but my issue is that only way to select a subset of testcases to run is to define a pattern that exists in the testcase name. This means I would still have to hard code a list of patterns to define what testcases to run. So if I have to hard code this list, what is the point of using the auto-discovery (please note this is rhetorical question)?

My end goal is to have a generic way to select which unittests to run during execution, in the form of check boxes or a text field the user can edit. Ideally the solution would be using Python 2.7 and need would need to run on Windows, OSX, and Linux.

Edit
To help clarify, I do not want the tool to generate the list of choices or the check boxes. The tool ideally would return a list of all of the tests in a directory, including the full path and what Suite (if any) the testcase belongs down. With this list, I would build the check boxes or a combo box the user interacts with and pass these tests into a testsuite on the fly to run.

Example Testcase

test_measure_5v_reference 
    1) Connect to DC power supply via GPIB
    2) Set DC voltage to 12.0 V
    3) Connect to a Digital Multimeter via GPIB
    4) Measure / Record the voltage at a given 5V reference test point
    5) Use unittest's assert functions to make sure the value is within tolerance
like image 795
Adam Lewis Avatar asked Nov 13 '22 19:11

Adam Lewis


1 Answers

Store each subset of tests in its own module. Get a list of module names by having the user select them using, as you stated, checkboxes or text entry. Once you have the list of module names, you can build a corresponding test suite doing something similar to the following.

testsuite = unittest.TestSuite()
for module in modules:
  testsuite.addTest(unittest.defaultTestLoader.loadTestsFromModule(module))
like image 77
D Krueger Avatar answered Dec 18 '22 11:12

D Krueger