Finding useless unit tests with PIT

Question

Assume we have a code we'd like to test:

class C {
    int doSmth() {
        return 1;
    }
}

Now assume we have 2 unit tests placed within a single class. The 1st one "test everything" while the 2nd one "does nothing":

@RunWith(JUnit4.class)
public final class CTest {

    @Test
    @SuppressWarnings("static-method")
    public void testDoSmth() {
        assertEquals(1, new C().doSmth());
    }

    @Test
    @SuppressWarnings("static-method")
    public void testDoSmth2() throws Exception {
        Thread.sleep(1000);
    }
}

This is an IRL example: I've seen dozens of tests "fixed" by replacing the test contents with some useless code, as the contract of code being tested changes over time.

Now, PIT "entry" unit is a class containing test methods (not an individual test metod itself), so in the above case PIT will not only show 100% line coverage, but also 100% mutation coverage.

Okay, I'm relieved to know I have 100% mutation coverage, but how do I identify a useless test -- testDoSmth2() in the above case (provided my mutation coverage is high)?

Answer 1

There is nothing currently built in to pitest, but the data you need to detect useless (in terms of detecting faults) tests is there.

The XML report outputs the killing test for each mutation (often at the level of a test method). Any test that does not kill a mutation could be removed without affecting the mutation score.

Of course tests that don't kill a mutation may still be valuable for other reasons e.g. describing what a unit does.

The extreme case shown in your example would however be more efficiently detected by static analysis - the test clearly doesn't exercise any code so can't possibly detect faults in it

Answer 2

I developed a manual method https://www.codeproject.com/Articles/4051293/Unit-Test-Suite-Quality-Estimation PIT can give you some hints though.