How can I write a unit test assertion that checks for an error with specified identifier and with a specific message?

Question

I'm using the new unit testing framework for MATLAB that was introduced in R2013a, matlab.unittest. I would like to write an assertion that both of the following things happen:

1. An exception with a specified identifier is raised.
2. The message of that exception satisfies some condition.

I've found the verifyError method but that only appears to allow me to check the identifier, or the error metaclass.

The other option appears to be verifyThat with the Throws constraint. That seems more promising but the documentation of Throws appears somewhat sparse and I cannot work out how to make it do what I want.

I realise that I could add the message text to the error identifier but I don't really want to do that. The message text comes from a native library that is called using a mex file. And the text is formatted with spaces and so on. The text could be quite long and would make a mess of the error identifier.

So, is it possible to achieve what I want, and if so how?

Answer 1

There is no ready function to do that. Here is one way to hack it.

Consider the following function we are testing. It throws a specific error on non-numeric inputs:

increment.m

function out = increment(x)
    if ~isa(x,'numeric')
        error('increment:NonNumeric', 'Input must be numeric.');
    end
    out = x + 1;
end

Here is the unit-test code:

IncrementTest.m

classdef IncrementTest < matlab.unittest.TestCase
    methods (Test)
        function testOutput(t)
            t.verifyEqual(increment(1), 2);
        end
        function testClass(t)
            t.verifyClass(increment(1), class(2));
        end
        function testErrId(t)
            t.verifyError(@()increment('1'), 'increment:NonNumeric');
        end

        function testErrIdMsg(t)
            % expected exception
            expectedME = MException('increment:NonNumeric', ...
                'Input must be numeric.');

            noErr = false;
            try
                [~] = increment('1');
                noErr = true;
            catch actualME
                % verify correct exception was thrown
                t.verifyEqual(actualME.identifier, expectedME.identifier, ...
                    'The function threw an exception with the wrong identifier.');
                t.verifyEqual(actualME.message, expectedME.message, ...
                    'The function threw an exception with the wrong message.');
            end

            % verify an exception was thrown
            t.verifyFalse(noErr, 'The function did not throw any exception.');
        end
    end
end

The use of the try/catch block is inspired by assertExceptionThrown function from the old xUnit Test Framework by Steve Eddins. (Update: the framework appears to have been removed from the File Exchange, I guess to encourage using the new built-in framework instead. Here is a popular fork of the old xUnit if you're interested: psexton/matlab-xunit).

If you want to give a bit more flexibility in testing the error message, use verifyMatches instead to match against strings using regular expressions.

---

Also if you are feeling adventurous, you could study the matlab.unittest.constraints.Throws class and create your own version which checks the error message in addition to the error id.

ME = MException('error:id', 'message');

import matlab.unittest.constraints.Throws
%t.verifyThat(@myfcn, Throws(ME));
t.verifyThat(@myfcn, ThrowsWithId(ME));

where ThrowsWithId is your extended version

---

EDIT:

ok so I went through the code of matlab.unittest.constraints.Throws and I implemented a custom Constraint class.

The class is similar to Throws. It takes an MException instance as input, and checks whether the function handle being tested throws a similar exception (checking both the error id and message). It can be used with any of the assertion methods:

• testCase.assertThat(@fcn, ThrowsErr(ME))
• testCase.assumeThat(@fcn, ThrowsErr(ME))
• testCase.fatalAssertThat(@fcn, ThrowsErr(ME))
• testCase.verifyThat(@fcn, ThrowsErr(ME))

To create a subclass from the abstract class matlab.unittest.constraints.Constraint, we must implement the two functions of the interface: satisfiedBy and getDiagnosticFor. Also note that we instead inherit from another abstract class FunctionHandleConstraint since it provides a few helper methods to work with function handles.

The constructor takes the expected exception (as an MException instance), and an optional input specifying the number of output arguments with which to invoke the function handle being tested.

The code:

classdef ThrowsErr < matlab.unittest.internal.constraints.FunctionHandleConstraint
    %THROWSERR  Constraint specifying a function handle that throws an MException
    %
    % See also: matlab.unittest.constraints.Throws

    properties (SetAccess = private)
        ExpectedException;
        FcnNargout;
    end

    properties (Access = private)
        ActualException = MException.empty;
    end

    methods
        function constraint = ThrowsErr(exception, numargout)
            narginchk(1,2);
            if nargin < 2, numargout = 0; end
            validateattributes(exception, {'MException'}, {'scalar'}, '', 'exception');
            validateattributes(numargout, {'numeric'}, {'scalar', '>=',0, 'nonnegative', 'integer'}, '', 'numargout');
            constraint.ExpectedException = exception;
            constraint.FcnNargout = numargout;
        end
    end

    %% overriden methods for Constraint class
    methods
        function tf = satisfiedBy(constraint, actual)
            tf = false;
            % check that we have a function handle
            if ~constraint.isFunction(actual)
                return
            end
            % execute function (remembering that its been called)
            constraint.invoke(actual);
            % check if it never threw an exception
            if ~constraint.HasThrownAnException()
                return
            end
            % check if it threw the wrong exception
            if ~constraint.HasThrownExpectedException()
                return
            end
            % if we made it here then we passed
            tf = true;
        end

        function diag = getDiagnosticFor(constraint, actual)
            % check that we have a function handle
            if ~constraint.isFunction(actual)
                diag = constraint.getDiagnosticFor@matlab.unittest.internal.constraints.FunctionHandleConstraint(actual);
                return
            end
            % check if we need to execute function
            if constraint.shouldInvoke(actual)
                constraint.invoke(actual);
            end
            % check if it never threw an exception
            if ~constraint.HasThrownAnException()
                diag = constraint.FailingDiagnostic_NoException();
                return
            end
            % check if it threw the wrong exception
            if ~constraint.HasThrownExpectedException()
                diag = constraint.FailingDiagnostic_WrongException();
                return
            end
            % if we made it here then we passed
            diag = PassingDiagnostic(constraint);
        end
    end

    %% overriden methods for FunctionHandleConstraint class
    methods (Hidden, Access = protected)
        function invoke(constraint, fcn)
            outputs = cell(1,constraint.FcnNargout);
            try
                [outputs{:}] = constraint.invoke@matlab.unittest.internal.constraints.FunctionHandleConstraint(fcn);
                constraint.ActualException = MException.empty;
            catch ex
                constraint.ActualException =  ex;
            end
        end
    end

    %% private helper functions
    methods (Access = private)
        function tf = HasThrownAnException(constraint)
            tf = ~isempty(constraint.ActualException);
        end

        function tf = HasThrownExpectedException(constraint)
            tf = metaclass(constraint.ActualException) <= metaclass(constraint.ExpectedException) && ...
                strcmp(constraint.ActualException.identifier, constraint.ExpectedException.identifier) && ...
                strcmp(constraint.ActualException.message, constraint.ExpectedException.message);
        end

        function diag = FailingDiagnostic_NoException(constraint)
            import matlab.unittest.internal.diagnostics.ConstraintDiagnosticFactory;
            import matlab.unittest.internal.diagnostics.DiagnosticSense;
            subDiag = ConstraintDiagnosticFactory.generateFailingDiagnostic(...
                constraint, DiagnosticSense.Positive);
            subDiag.DisplayDescription = true;
            subDiag.Description = 'The function did not throw any exception.';
            subDiag.DisplayExpVal = true;
            subDiag.ExpValHeader = 'Expected exception:';
            subDiag.ExpVal = sprintf('id  = ''%s''\nmsg = ''%s''', ...
                constraint.ExpectedException.identifier, ...
                constraint.ExpectedException.message);
            diag = constraint.generateFailingFcnDiagnostic(DiagnosticSense.Positive);
            diag.addCondition(subDiag);
        end

        function diag = FailingDiagnostic_WrongException(constraint)
            import matlab.unittest.internal.diagnostics.ConstraintDiagnosticFactory;
            import matlab.unittest.internal.diagnostics.DiagnosticSense;
            if strcmp(constraint.ActualException.identifier, constraint.ExpectedException.identifier)
                field = 'message';
            else
                field = 'identifier';
            end
            subDiag =  ConstraintDiagnosticFactory.generateFailingDiagnostic(...
                constraint, DiagnosticSense.Positive, ...
                sprintf('''%s''',constraint.ActualException.(field)), ...
                sprintf('''%s''',constraint.ExpectedException.(field)));
            subDiag.DisplayDescription = true;
            subDiag.Description = sprintf('The function threw an exception with the wrong %s.',field);
            subDiag.DisplayActVal = true;
            subDiag.DisplayExpVal = true;
            subDiag.ActValHeader = sprintf('Actual %s:',field);
            subDiag.ExpValHeader = sprintf('Expected %s:',field);
            diag = constraint.generateFailingFcnDiagnostic(DiagnosticSense.Positive);
            diag.addCondition(subDiag);
        end

        function diag = PassingDiagnostic(constraint)
            import matlab.unittest.internal.diagnostics.ConstraintDiagnosticFactory;
            import matlab.unittest.internal.diagnostics.DiagnosticSense;
            subDiag = ConstraintDiagnosticFactory.generatePassingDiagnostic(...
                constraint, DiagnosticSense.Positive);
            subDiag.DisplayExpVal = true;
            subDiag.ExpValHeader = 'Expected exception:';
            subDiag.ExpVal = sprintf('id  = ''%s''\nmsg = ''%s''', ...
                constraint.ExpectedException.identifier, ...
                constraint.ExpectedException.message);
            diag = constraint.generatePassingFcnDiagnostic(DiagnosticSense.Positive);
            diag.addCondition(subDiag);
        end
    end

end

Here is an example usage (using the same function as before):

t = matlab.unittest.TestCase.forInteractiveUse;

ME = MException('increment:NonNumeric', 'Input must be numeric.');
t.verifyThat(@()increment('5'), ThrowsErr(ME))

ME = MException('MATLAB:TooManyOutputs', 'Too many output arguments.');
t.verifyThat(@()increment(5), ThrowsErr(ME,2))

UPDATE:

The internals of some of the classes have changed a bit since I posted this answer. I updated the code above to work with the latest MATLAB R2016a. See the revision history if you want the older version.