Mixing Absolute and Relative Tolerances in TestCase.verifyEqual

Question

I've got a few test cases written with the class-based unit testing framework where I'm making sure an analytic gradient calculating I'm doing is reasonably close to a numerically calculated version. The problem is that the scales of the gradient vary pretty widely for the test cases I'm running. This means that while I'd like to use the AbsTol Name-Value pair, the test will commonly fail on what I'd consider an acceptable margin of error in some cases while RelTol will also fail on what I consider some acceptable cases. For example,

expected = [1e7 1e-7];
actual = [1e7+1 1e-5];

AbsTol = abs(expected-actual)
RelTol = abs((expected-actual)./expected)

I'd consider the first element here to satisfy the test since it's very close in relative terms and the second element here to satisfy the test since it's close in absolute terms, but I see no way to test if either is satisfied for both elements. I'm aware that I could do the comparison outside of verifyEqual, but that seems to be at the consequence of losing the nice diagnostics included in the testing framework.

Is there some way to create a unit test where I can specify that I'm considering an element-by-element comparison where either AbsTol or RelTol is satisfied?

Answer 1

Yes you can get what you want directly from 'verifyEqual', you just need to specify both 'AbsTol' and 'RelTol'. This will give you an elementwise OR operation to determine pass/fail. Something like this:

>> expected = [1e7 1e-7];
>> actual = [1e7+1 1e-5];

>> testCase = matlab.unittest.TestCase.forInteractiveUse;

>> testCase.verifyEqual(actual,expected,'RelTol',1e-3,'AbsTol',1e-3)
Interactive verification passed.

>> testCase.verifyEqual(actual,expected,'AbsTol',1e-3)
Interactive verification failed.

---------------------
Framework Diagnostic:
---------------------
verifyEqual failed.
--> The values are not equal using "isequaln".
--> The error was not within absolute tolerance.
--> Failure table:
            Index     Actual     Expected    Error    RelativeError    AbsoluteTolerance
            _____    ________    ________    _____    _____________    _________________

            1        10000001    10000000    1        1e-07            0.001            

Actual double:
       1.0e+07 *

       1.000000100000000   0.000000000001000
Expected double:
       1.0e+07 *

       1.000000000000000   0.000000000000010
>> 


>> testCase.verifyEqual(actual,expected,'RelTol',1e-3)
Interactive verification failed.

---------------------
Framework Diagnostic:
---------------------
verifyEqual failed.
--> The values are not equal using "isequaln".
--> The error was not within relative tolerance.
--> Failure table:
            Index    Actual    Expected     Error     RelativeError    RelativeTolerance
            _____    ______    ________    _______    _____________    _________________

            2        1e-05     1e-07       9.9e-06    99               0.001            

Actual double:
       1.0e+07 *

       1.000000100000000   0.000000000001000
Expected double:
       1.0e+07 *

       1.000000000000000   0.000000000000010

Note that the default operation of verifyEqual when both 'AbsTol' and 'RelTol' is the same as the following:

>> import matlab.unittest.constraints.IsEqualTo;
>> import matlab.unittest.constraints.AbsoluteTolerance;
>> import matlab.unittest.constraints.RelativeTolerance;
>> testCase.verifyThat(actual, IsEqualTo(expected, ...
       'Within', AbsoluteTolerance(abstol) | RelativeTolerance(reltol)));

...but if you want to change that to an AND operation, so that all values must be within both tolerance types, you can do that as follows:

>> testCase.verifyThat(actual, IsEqualTo(expected, ...
       'Within', AbsoluteTolerance(abstol) & RelativeTolerance(reltol)));

Finally, one more thing that might be helpful is that the tolerance can be specified either as a single scalar value or as a value the same size as the values being compared. This allows you to have tolerance profiles that are different for each element. For example in your case you can make the absolute error 10 for the first element and 1e-3 for the second and it will pass with a single tolerance:

>> testCase.verifyEqual(actual,expected,'AbsTol',[10, 1e-3])
Interactive verification passed.