Short Question
Is it possible to re-attempt a unittest upon failure / error N number of times OR based on a predefined function. (Like a user's prompt)
Background
To avoid retyping an entire page of system information, please see SO question on passing data to unittests and on auto test discovery for more details on my physical set up.
Regarding the question at hand, I know I can do it by re-writting my test cases to loop until it gets the required results (see pseudo code below) then assert based off of this. However I rather not go and rewrite 100's of test cases.
I there will be someone that will point out that if a unittest fails, it should just fail and be done. I agree with this 100% if human error could be removed. This is a physical system that I am connected to and there are many times the leads from the digital multimeter are not connected well and it could fail because of a loose connection.
Pseudo Workaround
class Suite_VoltageRegulator(unittest.TestCase):
def test_voltage_5v_regulator(self):
keep_running = 'y'
error_detected = False
print '\n'
# Display User Test Configuration
msg = \
'1) Connect Front DMM GND(black) to the TP_COM\n' +\
'2) Connect Front DMM POS(red) to the TP-A\n' +\
'3) Ensure the DMM terminal button indicates FRONT'
continue_test = prompt.Prompt_Ok_Cancel('User Action Required!', msg)
if not continue_test:
self.assertTrue(False, 'User Canceled Test!')
while(keep_running == 'y'):
try:
# Run the test
results = measure_voltage_from_system()
# Analyze Results
test_status = pf.value_within_range(results, REGULATOR_5V_LOW, REGULATOR_5V_HIGH)
catch:
error_detected = True
# Retest Failed Cards
if(test_status == False):
keep_running = rawinput('Test FAILED: Retry Test? (y/n):')
elif(error_detected == True):
keep_running = rawinput('Test ERROR: Retry Test? (y/n):')
else:
keep_running = 'n'
# Inform the user on the test results
self.assertTrue(test_status, 'FAIL: 5V Regulator (' +str(results)+ ') Out of Range!')
EDIT 8/22/11 3:30 PM CST
I do know that I am violating the definition of the unittest in this use-case. These issues / comments are also addressed in a few of my other SO questions. One of the design goals we chose was to leverage an existing framework to avoid having to "reinvent the wheel". The fact that we chose python's unittest was not based on it's definition, but it's flexibility and robustness to execute and display a series of tests.
Going into to this project, I knew there would be some things that would require workarounds because this module was not intended for this use. At this point in time, I still believe that these workarounds have been easier / cheaper than rewriting my own test runner.
EDIT 8/22/11 5:22 PM CST
I am not dead set on using unittest for future projects in this manner, however I fairly set on using an existing frame work to avoid duplicating someone else's efforts. A comment below is an example of this pycopia-QA appears to be a good fit for this project. The only drawback for my current project is it is that I have written hundreds unittest test-cases, if I were to rewrite them it would be a very large undertaking (noting that it will also be a non-funded effort)
EDIT 8/24/11 11:00 AM CST
It may be clear for future projects to switch to a more tailored frame work for this type of testing. However I still have projects running with unittest so a solution using only unittest (or nose + 3rd addon) is still needed.
You wouldn't want to check the result body of a rest API if the result code is not the expected, for example. From the python documentation: assertEqual(first, second, msg=None) Test that first and second are equal. If the values do not compare equal, the test will fail.
Pytest has rich inbuilt features which require less piece of code compared to unittest. In the case of unittest, we have to import a module, create a class and then define testing functions inside the class. But in the case of pytest, we have to define the functions and assert the conditions inside them.
Alternate syntax for skipping test is using instance method skipTest() inside the test function.
4 years after the original question - I hope that anyone would care :) Here's my solution for doing this on top of unittest. It's kind of ugly and relies on the implementation of the TestCase base class , but it works.
class MyTest(unittest.TestCase):
###
### Insert test methods here
###
# Wrapping each test method so that a retry would take place.
def run(self, result=None):
self.origTestMethodName = self._testMethodName
self._testMethodName = "_testRetryWrapper"
super(MyTest, self).run(result)
self._testMethodName = self.origTestMethodName
def _testRetryWrapper(self):
testMethod = getattr(self, self.origTestMethodName)
retryAttemptsLeft = settings.testRetryCount
while True:
try:
testMethod()
break
except:
if retryAttemptsLeft == 0:
raise
else:
retryAttemptsLeft = retryAttemptsLeft - 1
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With