Python Pre-testing for exceptions when coverage fails

Question

I recently came across a simple but nasty bug. I had a list and I wanted to find the smallest member in it. I used Python's built-in min(). Everything worked great until in some strange scenario the list was empty (due to strange user input I could not have anticipated). My application crashed with a ValueError (BTW - not documented in the official docs).

I have very extensive unit tests and I regularly check coverage to avoid surprises like this. I also use Pylint (everything is integrated in PyDev) and I never ignore warnings, yet I failed to catch this bug before my users did.

Is there anything I can change in my methodology to avoid these kind of runtime errors? (which would have been caught at compile time in Java / C#?).

I'm looking for something more than wrapping my code with a big try-except. What else can I do? How many other build in Python functions are hiding nasty surprises like this???

Answer 1

The problem here is that malformed external input crashed your program. The solution is to exhaustively unit test possible input scenarios at the boundaries of your code. You say your unit tests are 'extensive', but you clearly hadn't tested for this possibility. Code coverage is a useful tool, but it's important to remember that covering code is not the same as thoroughly testing it. Thorough testing is a combination of covering usage scenarios as well as lines of code.

The methodology I use is to trust internal callers, but never to trust external callers or input. So I explicitly don't unit test for the empty list case in any code beyond the first function that receives the external input. But that input function should be exhaustively covered.

In this case I think the library's exception is reasonable behaviour - it makes no sense to ask for the min of an empty list. The library can't legitimately set a value such as 0 for you, since you may be dealing with negative numbers, for example.

I think the empty list should never have reached the code that asks for the min - it should have been identified at input, and either raised an exception there, or set it to 0 if that works for you, or whatever else it is that does work for you.