Sonar rule S2699: Not all asserts are recognized as valid assertions

Question

We are running Sonarqube 5.6.1 with the Java Plugin 4.1 and having some troubles using the Sonar rule S2699 (Test should include assertions).

Using this example test class

import mypackage.Citit1543Dummy;
import mypackage.Citit1543OtherDummy;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.mockito.MockitoAnnotations;

import java.util.Arrays;

import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.isIn;
import static org.hamcrest.Matchers.lessThan;
import static org.hamcrest.core.IsNot.not;
import static org.mockito.Matchers.notNull;
import static org.mockito.Mockito.mock;
import static org.mockito.Mockito.times;
import static org.mockito.Mockito.verify;
import static org.junit.Assert.assertThat;

public class Citit1543Test {

    @Before
    public void setUp() {
        MockitoAnnotations.initMocks(this);
    }

    @Test
    public void test1() {
        assert true;
    }

    @Test
    public void test2() {
        Assert.assertTrue(1 > (2-3));
    }

    @Test
    public void test3() {
        Assert.assertFalse(1 > (100-1));
    }

    @Test
    public void test4() {
        Assert.assertThat("test", 1, is(1));
    }

    @Test
    public void test5() {
        Assert.assertArrayEquals(new String[0], new String[0]);
    }

    @Test
    public void test6() {
        Assert.assertEquals(1 > 0, true);
    }

    @Test
    public void test7() { // asserts in another method
        test7asserts(1, 1);
    }

    private void test7asserts(int a, int b) {
        Assert.assertTrue(a == b);
    }

    @Test
    public void test8() {
        test8asserts(1, 2);
    }

    private void test8asserts(int a, int b) {
        Assert.assertNotSame(a, b);
    }

    @Test
    public void test9() {
        Citit1543Dummy dummy = new Citit1543Dummy();
        dummy.otherDummy = mock(Citit1543OtherDummy.class);
        dummy.doSomething();
        verify(dummy.otherDummy, times(1)).doSomething();
    }

    @Test
    public void test10() {
        Citit1543Dummy dummy = new Citit1543Dummy();
        dummy.otherDummy = mock(Citit1543OtherDummy.class);
        dummy.doSomething();
        test10verifies(dummy.otherDummy);
    }

    private void test10verifies(Citit1543OtherDummy otherDummy) {
        verify(otherDummy, times(1)).doSomething();
    }

    @Test
    public void test11() {
        Assert.assertThat("test", "", not(1));
    }

    @Test
    public void test12() {
        Assert.assertThat("test", 1, lessThan(2));
    }

    @Test
    public void test13() {
        Long[] arr = new Long[] { 1L, 2L, 3L, 4L };
        assertThat("Just testing", arr, is(new Long[] {
            1L, 2L, 3L, 4L
        }));
    }
}

our Sonarqube instance flags the test cases test1 (assert statement not recognized), test7 (assert statements in another method), test8 (same) , test10 (Mockitos verify in another method), test11 and test13 as methods without assertions. I'm pretty sure that there are a lot more methods which aren't recognized (yes, unfortunately we use a bunch of different mocking/testing framework across our projects).

For now, we started to //NOSONAR whenever one of the asserts/verifies aren't recognized. Is there an easy way to include these methods to be recognized as valid asserts?

Answer 1

Many of your stated issues are known and indeed (in some form of another) marked as FP: test1: The current flow analysis ignores assert statements. See this post over at the groups.

The cases test7, test8 and test10 are related to the lack of not having cross-procedural analysis: They are valid cases but the current flow doesn't know that (ex.) test7assert is a valid assert statement for another method. See this post over at the groups.

Your other cases also produce false positives in the tests of S2699. I'd expect that once a SonarSource dev reads this topic that they'll create a ticket to resolve the cases in test11/13. But as I'm not a dev of them I can't guarantee that of course.

As to :

Is there an easy way to include these methods to be recognized as valid asserts?

No, the valid assertions are defined within the code of S2699 and are not a parameter. Some of your cases will require a more complex flow analysis whilst the last couple just seem to boil down to some missing definitions or too strict definitions, but I didn't deep-dive into the reasons why they produce FPs.