Why might rdmd not be running all unit tests?

Question

Unfortunately I cannot seem to reproduce this behavior in a minimal working example, so this may be too vague. However, I can at least state what is not causing the behavior.

I have a D module containing several classes with unit tests, with structure similar to:

import std.stdio;

class c1{
        int c1func(int blah){ return blah; }
        unittest{
                writeln("Testing c1 func.");
                stdout.flush();
        }
}

class c2(T:c3) : c1{
        private static int c2func(int blah){ return blah; }
        unittest{
                writeln("Testing c2 func.");
                stdout.flush();
        }

        int c2fun2(int blah){
                T tmp = new T();
                return tmp.c3fun(blah);
        }
}

class c3{
        abstract int c3fun(int blah);
}

class c4 : c3{
        override int c3fun(int blah){ return blah; }
}

unittest{
        writeln("Testing class c1.");
        stdout.flush();
        c1 myc2 = new c2!(c4)();
}

My expectation is that a call of the form:

rdmd --main -unittest tmp.d

will produce output

Testing c1 func.
Testing class c1.
Testing c2 func.

and it does (c2 func's unit tests are not run until the time of instantiation).

However, in my similar, but much longer D module, no unit tests for the template class c2's counterpart are run (I've put print statements identical to the one's here throughout my other unit tests to be sure).

This is a bit worrisome, as in fact, I'd been relying on the correctness of these other unit tests for some time. Further, rdmd happily picks up on syntax errors in the unit test code, despite never running said code.

So here's what I can rule out:

• The unit tests are being read by rdmd.
• rdmd is actually being run against the most recent version of the file.
• rdmd apparently is happy to run unit tests for implementations of template classes, even with specializations of abstract classes for their arguments, as long as those template classes are actually instantiated.
• The functions in my template class (akin to c2 in the example code) are actually getting called during the main unit test block at the bottom, so the template class has definitely been instantiated.
• rdmd itself is not at fault, as compiling using dmd and the same flags also fails to run these unit tests.

Any ideas about what could be going wrong here? I'd provide a MWE if I could, but every version of one I've written seems to work fine!

Answer 1

It seems that this results from the way that rdmd orders the unit tests, which is not in the way I'd expected.

It appears that RDMD calls the unit tests for an instantiated template class not at the time of instantiation, but instead immediately after the unit test in which the class is instantiated has completed.

Here's a MWE:

import std.stdio;

class c1{
        int c1func(int blah){ return blah; }
        unittest{
                writeln("Testing c1 func.");
                stdout.flush();
        }
}

class c2(T:c3) : c1{
        private static int c2func(int blah){ return blah; }
        unittest{
                writeln("Testing c2 func.");
                stdout.flush();
        }

        int c2fun2(int blah){
                T tmp = new T();
                return tmp.c3fun(blah);
        }
}

class c3{
        abstract int c3fun(int blah);
}

class c4 : c3{
        override int c3fun(int blah){ return blah; }
}

unittest{
        writeln("Testing class c1.");
        stdout.flush();
        c1 myc2 = new c2!(c4)();
        assert(1==0);
}

In this case, the unit test at the bottom will fail (because of assert(1==0)), and so the template class's unit tests are never run.

This poses a major problem when using template classes together with unit tests. If your template class fails is broken in some way that causes the unit test where it was instantiated to fail, then its own unit tests are never run. In the case of silent failure (a silent exit was triggered in my code by bugs in the template class), this will manifest as the template class's tests never running.

Summary: DMD usually just runs unit tests in the order that they appear. However, since template classes don't exist without some completion of the template, unit tests inside a template class are not run in the order that they appear. Instead, they are run (in order), immediately after the first unit test where the template class is instantiated.

Workaround: adding a separate unit test immediately below the template class that just instantiates it with all the types one wants to test will cause the unit tests to be run, in the correct order. However, a seemingly better policy would be for to run those unit tests before the test block in which the instantiation occurs!