How to validate a unit test with random values

Question

How to validate a unit test with random values? I need guarantee that gen_age returns an integer between 15 and 99, but this code is not correct.

import random
import unittest


def gen_age():
    # generate integer between 15 and 99
    return random.randint(15, 99)


class AgeTest(unittest.TestCase):

    def setUp(self):
        self.a = gen_age()

    def test_choice(self):
        element = random.choice(self.a)
        self.assertTrue(element in self.a)

    def test_sample(self):
        for element in random.sample(self.a, 98):
            self.assertTrue(element in self.a)

if __name__ == '__main__':
    unittest.main()

Answer 1

I think there is more to build on top of the answer from @user983716 because:

1. that answer changes the way the source code works by forcing a seed, and the real source is non-deterministic
2. the test can do much more to document the expected behavior, while that test is obscure and probably confusing to most

The real desire is not to test the functionality of the Random.randint, but rather that the output is correct. This was suggested by @rufanov, though that solution could be improved.

Throughout this answer, we assume that the implementation is in a separate package.file from the tests.

Let's start with the following:

import unittest

from package.file import get_age

class AgeTest(unittest.TestCase):

    def test_gen_age_generates_a_number_between_15_and_99(self):
        age = gen_age()
        self.assertGreaterEqual(age, 15)
        self.assertLessEqual(age, 99)

That's a good test to begin with, because it provides clear output if a failure occurs:

AssertionError: 14 not greater than or equal to 15
AssertionError: 100 not less than or equal to 99

Okay, but we also want to make sure it is a random number, so we can add another test that ensures that we get it from randint as expected:

@unittest.mock.patch('package.file.random')
def test_gen_age_gets_a_random_integer_in_the_range(self, mock_random):
    gen_age()
    mock_random.randint.assert_called_with(15, 99)

We do two important things here:

• The random object (from the file in which gen_age is defined) is patched so that we can perform testing against it without having to rely upon the real implementation
• A single assertion is made confirming that the expected 15 and 99 arguments are provided to randint, so that the right range is given

Additional tests could be written that assert against the real return value, confirming that the number given is always the randomized one. This would provide confidence that the method is directly returning the randomized value, as it could be conceivable for the method to do something more, or even return some arbitrary value, even if it internally still makes the randint call.

For example, let's say someone changed gen_age() as follows:

def gen_age():
    age = random.randint(15, 99) # still doing what we're looking for
    return age + 1               # but now we get a result of 16-100

Uh oh, now for only those cases where randint returns 99 will our first test fail, and the second test will still pass. This is a production error waiting to happen...

A simple, but effective way to confirm the result, then, might be as follows:

@unittest.mock.patch('package.file.random')
def test_returns_age_as_generated(mock_random):
    mock_random.return_value = 27
    age = get_age()
    self.assertEqual(age, 27)

There is one final flaw here, though... What if the value returned is changed to this:

def gen_age():
    age = random.randint(15, 99)
    return 27

Now all the tests are passing, but we still aren't getting the randomized result we really want. To fix this we need to randomize the test value too...

It turns out that our source code points to the answer - just use the real implementation as part of that second test. For this, we first need to import the original random:

from package.file import get_age, random

And then we'll modify the last test we wrote, resulting in this:

@unittest.mock.patch('package.file.random')
def test_returns_age_as_generated(mock_random):
    random_age = random.randint(15, 99)
    mock_random.randint.return_value = random_age
    age = get_age()
    self.assertEqual(age, random_age)

Thus, we would end up with the following full suite of tests:

import unittest

from package.file import get_age, random


class AgeTest(unittest.TestCase):

    def test_gen_age_generates_a_number_between_15_and_99(self):
        age = gen_age()
        self.assertGreaterEqual(age, 15)
        self.assertLessEqual(age, 99)

    @unittest.mock.patch('package.file.random')
    def test_gen_age_gets_a_random_integer_in_the_range(self, mock_random):
        gen_age()
        mock_random.randint.assert_called_with(15, 99)

    @unittest.mock.patch('package.file.random')
    def test_returns_age_as_generated(mock_random):
        random_age = random.randint(15, 99)
        mock_random.randint.return_value = random_age
        age = get_age()
        self.assertEqual(age, random_age)