Test coverage for many AND/OR conditions in one statement

Question

Example statement:

if (conditionA && conditionB && conditionC && conditionD) {
    return true;
}

I could write unit tests for all 2^4 combinations, but that would easily go out of hand if more conditions are added.

What should be my unit testing strategy to cover all conditions for a statement like this? Is there any other way to make the code more robust?

Answer 1

As an Intro to my answer, I would like to explain again, why we do software testing. There is a really big misunderstanding by most test folks.

• We do NOT test software to show that it is error free (That is impossible. Already for minor complex software)
• We perform a “constructive test”, to prove that the functionality is acceptable and requirements or features are “working”.
• And most important: We try to find as much as possible errors (“Destructive test”). We will never find all errors. SRGM can be applied to show, how deep testing should go.

Then, and this is already an answer to a part of your question “What should be my unit testing strategy?”

I will cite Automotive SPICE (PAM 3.1), a well-known and proven process model, Process SWE.4, Software Unit Verification:

“The purpose of the Software Unit Verification Process is to verify software units to provide evidence for compliance of the software units with the software detailed design and with the non-functional software requirements.”

Another set of unit test descriptions, for software with higher demands for quality and especially safety, can be taken from ISO 26262 “Road vehicles — Functional safety —”, Part 6: Product development at the software level”, chapter 9, tables 10, 11, 12

Methods for software unit testing

Requirements-based test
Interface test
Fault injection test
Resource usage test
Back-to-back comparison test between model and code, if applicable

Methods for deriving test cases for software unit testing

Analysis of requirements
Generation and analysis of equivalence classes
Analysis of boundary values
Error guessing

And now the most important, answering the second part of your question, you should do a structural coverage analysis (NOT a structural test), to evaluate the completeness of test cases and to demonstrate that there is no unintended functionality. Do never mix up structural test and structural coverage.

So, you will test and check, if the requirements are covered and you will do a structural coverage measurement to prove that. If the coverage result is too low, then you will add more test cases.

The recommended coverage metric is MCDC.

Of course, you can select also one of the many others Coverage methods. But then you should give a rationale in your test strategy, why you do this.

Looking at your question again:

if (conditionA && conditionB && conditionC && conditionD) 
{
    return true;
}

It seems that you are asking for a recommendation for a structural test. I will also answer this question, but be aware that with this method, you are just testing, if the complier works correctly.

For the boolean expression at hand (and other more complex expressions), you might never find one of the following errors:

Error classes

Expression Negation Fault (ENF)
Sub-Expression Negation Fault (SENF)
Sub-Expression Omission Fault (SEOF)
Literal Negation Fault (LNF)
Literal Omission Fault (LOF)
Literal Reference Fault (LRF)
Literal Insertion Fault (LIF)
Operator Reference Fault (ORF)
Stuck-at-1 Fault (SA1) 
Stuck-at-0 Fault (SA0)
Parenthesis Insertion Fault (PIF)
Parenthesis Omission Fault (POF)
Parenthesis Shift Fault (PSF)

Remember, what I said in the beginning, testing should find errors (destructive test). Otherwise you will maybe miss the above-mentioned errors.

Example:

If your requirements originally intended to have “OR”s, instead of “AND”s in your expression (Error class ORF), using Condition Coverage, with test vector “TTTT” and “FFFF”, will give you 100% condition coverage and 100% decision or branch coverage. But you will not find the bug. MCDC would reveal the problem.

You also mentioned the possibility to test all combinations (MCC, Multiple Condition Coverage). For 4 variables, this is OK. But for more variables the test execution duration will grow geometrically. This is not manageable. And that was one of the reasons, why MCDC has been defined.

Now, lets assume that your example statement is correct and coming back to the definition for test cases for a structural test, based on MCDC, for your expression.

There are several definitions available, mostly talking about “Unique Cause” MCDC, neglecting the fact, that meanwhile also “Masking MCDC” and “Unique cause + Masking MCDC” are certified and approved criterions. For those you need to forget about all the tutorials starting with a BlackBox view on a truth table. Talking about structural coverage or test, it should be clear, that only a WhiteBox view will work. And if you happen to develop in a language with boolean short cut evaluation (like for example in Java, C or C++), it will even be more obvious that a WhiteBox view is mandatory.

---

For your boolean expression/decision (“abcd”), and applying boolean short cut evaluation, there are overall 16 Unique Cause MCDC test pairs:

1   Influencing Condition: 'a'  Pair:  0, 15   Unique Cause
2   Influencing Condition: 'a'  Pair:  1, 15   Unique Cause
3   Influencing Condition: 'a'  Pair:  2, 15   Unique Cause
4   Influencing Condition: 'a'  Pair:  3, 15   Unique Cause
5   Influencing Condition: 'a'  Pair:  4, 15   Unique Cause
6   Influencing Condition: 'a'  Pair:  5, 15   Unique Cause
7   Influencing Condition: 'a'  Pair:  6, 15   Unique Cause
8   Influencing Condition: 'a'  Pair:  7, 15   Unique Cause
9   Influencing Condition: 'b'  Pair:  8, 15   Unique Cause
10  Influencing Condition: 'b'  Pair:  9, 15   Unique Cause
11  Influencing Condition: 'b'  Pair: 10, 15   Unique Cause
12  Influencing Condition: 'b'  Pair: 11, 15   Unique Cause
13  Influencing Condition: 'c'  Pair: 12, 15   Unique Cause
14  Influencing Condition: 'c'  Pair: 13, 15   Unique Cause
15  Influencing Condition: 'd'  Pair: 14, 15   Unique Cause

Resulting in a recommended MCDC test set (There are more than one solutions):

Test Pair for Condition 'a':    0  15   (Unique Cause)
Test Pair for Condition 'b':    8  15   (Unique Cause)
Test Pair for Condition 'c':   12  15   (Unique Cause)
Test Pair for Condition 'd':   14  15   (Unique Cause)

Test vector: Final Result: 0 8 12 14 15

 0:  a=0  b=0  c=0  d=0    (0)
 8:  a=1  b=0  c=0  d=0    (0)
12:  a=1  b=1  c=0  d=0    (0)
14:  a=1  b=1  c=1  d=0    (0)
15:  a=1  b=1  c=1  d=1    (1)

---

Without boolean short cut evaluation, you have, very obviously, only 4 Unique Cause MCDC test pairs:

1  Influencing Condition: 'a'  Pair:  7, 15   Unique Cause
2  Influencing Condition: 'b'  Pair: 11, 15   Unique Cause
3  Influencing Condition: 'c'  Pair: 13, 15   Unique Cause
4  Influencing Condition: 'd'  Pair: 14, 15   Unique Cause

Leading to one deterministic solution:

Test vector: Final Result: 7 11 13 14 15

 7:  a=0  b=1  c=1  d=1    (0)
11:  a=1  b=0  c=1  d=1    (0)
13:  a=1  b=1  c=0  d=1    (0)
14:  a=1  b=1  c=1  d=0    (0)
15:  a=1  b=1  c=1  d=1    (1)

---

I hope, I could shed some more light on the issue.

If you want to explore MCDC in more detail, with tool support, you may have a look into

MCDC