Menu
If my unit test fails, but the underlying code actually works, would this be called a false positive, or a false negative?
(Here is the context. I run a bunch of tests in parallel and some tests fail. I run them sequentially and they all pass. All the code actually works, there is just an issue with the parallel test runner.)
My question is just with the nomenclature though. I've found examples of people calling it either. After reading Wikipedia, I would've thought it is a false positive, but notably Andrew Ng in his machine learning class said:
If the test passes, even if the code is broken, that is a false positive. If the test fails, when the code is NOT broken, that is a false negative.
259
In the context of automated software testing, a False Positive means that a test case fails while the software under test does not have the bug in which the test tries to catch. As a result of a false positive, test engineers spend time hunting down a bug that does not exist.
Since false-negative results pose greater risks, most testing applications are set up to minimise the occurrence of false-negative results. This means that false-positive results are more likely to occur and are therefore more often found as a topic of discussion.
A false positive is an error in binary classification in which a test result incorrectly indicates the presence of a condition (such as a disease when the disease is not present), while a false negative is the opposite error, where the test result incorrectly indicates the absence of a condition when it is actually ...
Most home pregnancy tests are reliable, for example Clearblue's tests have an accuracy of over 99% from the day you expect your period, and while it's possible a test showing a negative result is wrong, particularly if you're testing early, getting a false positive is extremely rare.
I believe it's the other way around than what the other answers are saying.
At least that's how false positives and false negatives are defined in the "XUnit Test Patterns: Refactoring Test Code" book by Gerard Meszaros.
An easy way to understand this is to think of it like in medicine where the tests are "testing for diseases".
If you have the disease you are "disease-positive".
In our software world, you can think it like this:
A POSITIVE test, means you are POSITIVE to a bug i.e. Test fails because you are "NullReferenceExceptionitis-positive"
A NEGATIVE test, means you are NEGATIVE to a bug i.e. Test passes because you are "StackOverflu-negative"
So, keep in mind bug = disease and:
A FALSE positive (to a bug) means your code is falsely accused to have a bug. (Your code has no bugs, yet test fails)
A FALSE negative (to a bug) means your code is falsely declared to have no bugs. (Your code has bugs, yet test passes)
A TRUE positive means your code is rightfully(truly) accused to have bugs. (Your code has bugs, test fails)
A TRUE negative means your code is rightfully(truly) declared to have no bugs (Your code has no bugs, test passes)
I hope this helps.
References: http://xunitpatterns.com/false%20positive.html
And from that book:
If we are having problems with Buggy Tests or Production Bugs, we can reduce the risk of false negatives (tests that pass when they shouldn’t)
also:
false negative
A situation in which a test passes even though the system under test (SUT) is not working properly. Such a test is said to give a false-negative indication or a “false pass.” See also: false positive.
false positive
A situation in which a test fails even though the system under test (SUT) is working properly
194
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
