Is there some kind of 'assertion' coverage tool (for Java)?

Question

Before this question is marked as duplicate, please read it. ;) There are already several questions about coverage tools and such, however this is a bit different than the usual ones (I hope).

According to wikipedia there are several different kind of 'coverage' variations that affect several different aspects of the term 'coverage'.

Here a little example:

public class Dummy {
    public int a = 0;
    public int b = 0;
    public int c = 0;

    public void doSomething() {
        a += 5;

        b += 5;

        c = b + 5;
    }
}

public class DummyTest {

    @Test
    public void testDoSomething() {
        Dummy dummy = new Dummy();

        dummy.doSomething();

        assertEquals( 10, dummy.c );
    }
}

As you can see, the test will have a coverage of 100% lines, the assertion on the value of field 'c' will cover this field and indirectly also cover field 'b', however there is no assertion coverage on field 'a'. This means that the test covers 100% of the code lines and assures that c contains the expected value and most probably also b contains the correct one, however a is not asserted at all and may a completely wrong value.

So... now the question: Is there a tool able to analyze the (java) code and create a report about which fields/variables/whatever have not been (directly and/or indirectly) covered by an assertion?

(ok when using getters instead of public fields you would see that getA() is not called, but well this is not the answer I'd like to hear ;) )

Answer 1

There are hundreds of definition of "test coverage", of which the COTS tools only handle a very few at best. (My company builds test coverage tools so we track this kind of thing). See this lecture on test coverage for an interesting overview.

The closest definition I have heard is one for data coverage; depending on your definition :-{ it tells you that each data item has been written and read during execution. The lecture talks about verifying that every write and every read has been exercised as a special case.

I don't know the hundreds of definitions by heart, but you may have invented yet one more: data coverage restricted to assertions.

Answer 2

As you can see, the test will have a coverage of 100% lines, the assertion on the value of field 'c' will cover this field and indirectly also cover field 'b', however there is no assertion coverage on field 'a'. This means that the test covers 100% of the code lines and assures that c contains the expected value and most probably also b contains the correct one, however a is not asserted at all and may a completely wrong value.

Well, "cover" unfortunately means different things to different people... This test indeed exercises 100% of the code lines, but it does not test them all.

What you're looking for is handled well by mutation testing.

Have a look at Jester, which uses mutation testing to report on code coverage.