What is software fault injection really used for?

Question

My basic understanding of software fault injection is as follows:

One can't run all possible tests to test a software. So one resorts to make modifications in the code and analyzing the degree of errors that are produced from it.

But how is this useful?

Like say we had:

function foo(a, b) {
   return a/b;
}

and we changed it to

function foo(a, b) {
  return Math.floor(a/b);
}

So what of it?

How was this useful at all?

EDIT

@Leo, Say I wrote a software that finds Fibonacci numbers. I write test that look like this: assert(fib(1) == 1);

assert(fib(0) == 0);

assert(fib(3) == 2);

I claim 100% coverage since all lines of code are executed here.

My client runs these tests and all of them pass. So he thinks, "maybe the tests are themselves wrong. let me introduce some changes in them".

So he changes one of them to assert(fib(1) == 5);

and the test fails. What can he conclude from it?

Answer 1

Suppose you've hired a company to deliver to you some software and they promised they unit test their code with 90% coverage, which seems to be a great deal.

So you decide to insert errors in these tests, so you expect to see a much lower coverage of passing tests, but, well, after inserting the errors, you discover that it still 90% coverage :-)

How useful are these tests?

for example, is this test right?

@Test
public void testAdd() {
    int result = 0;
    Claszz c = new Claszz();
    int result2 = c.add(-1, 1);
    assertEquals(result,0);
}

Answer 2

Software fault injections are basically used to check the code coverage done by the tests. If a set of test data is run on the code and all of them pass it would definitely not mean that the code is 100% bug free. One reason might be that given the data, some part of the code is not reachable. In this case, the code is modified to see if it still works as expected.

This is primarily used to test the exceptions or error handling which might be encountered rarely.

For example, if there a field which only accepts positive number and there is a requirement that on entering negative values error should be displayed.

In this case the property of the field can be set such that the user is not able to enter - in it. So the condition of negative value is not testable.

Here the code will be modified so that the user can enter -ve value and check the error message.

Answer 3

Well this is an old question but Google got me here so I will just add my views.

Software fault injection and the related estimates are related to test coverage, but I would say not simply as a percentage of code. It is a measure of how good your tests are at uncovering faults. So you inject 100 faults and your tests find 60 of those. You missed 40% of the faults. You might have had 100% code coverage in your tests and still missed those faults. If you had a set of tests all saying assert(1 == 1), you would have missed 100% of your faults.

Of course, there is then the question of the quality of the injections (i.e., do they trigger observable faults, is the fault important enough to test, is the code reachable, etc). But that is another matter.

Finally, another aspect is robustness testing, with tools like Chaos Monkey that kills VM instances from your live system, aiming to evaluate how your system handles failures, recovery, etc. But this is a slightly different viewpoint since you inject faults at the system level, and I guess this question was on code/unit test level.