About the right way to use Julia & Base.Test: run all tests even if some fail

Question

Using Base.Test for my unit tests, I am surprised by immediate exit right after the first test failure.

Let's consider this runtest.jl file:

using Base.Test

@testset "First" begin
    # test fails
    @test false
end;

@testset "Second" begin
    # never run... 
    @test true 
end;

The output of julia runtest.jl is always (the second test is never run):

First: Test Failed
  Expression: false
Stacktrace:
 [1] macro expansion at /home/picaud/Temp/runtests.jl:14 [inlined]
 [2] macro expansion at ./test.jl:860 [inlined]
 [3] anonymous at ./<missing>:?
Test Summary: | Fail  Total
First         |    1      1
ERROR: LoadError: Some tests did not pass: 0 passed, 1 failed, 0 errored, 0 broken.

My question: how to run and report all test results even if some tests fail?

Answer 1

Reading the Julia doc Working-with-Test-Sets it seems one must systematically use nested testset:

Typically a large number of tests are used to make sure functions work correctly over a range of inputs. In the event a test fails, the default behavior is to throw an exception immediately. However, it is normally preferable to run the rest of the tests first to get a better picture of how many errors there are in the code being tested.

and later this quote:

The @testset() macro can be used to group tests into sets. All the tests in a test set will be run, and at the end of the test set a summary will be printed.

In the previous peculiar example, this

using Base.Test

@testset "All tests" begin

    @testset "First" begin
        @test false 
    end;

    @testset "Second" begin
        # is run, ok
        @test true 
    end;

end;

will run all tests:

First: Test Failed
  Expression: false
Stacktrace:
 [1] macro expansion at /home/picaud/Temp/runtests.jl:5 [inlined]
 [2] macro expansion at ./test.jl:860 [inlined]
 [3] macro expansion at /home/picaud/Temp/runtests.jl:4 [inlined]
 [4] macro expansion at ./test.jl:860 [inlined]
 [5] anonymous at ./<missing>:?
Test Summary: | Pass  Fail  Total
All tests     |    1     1      2
  First       |          1      1
  Second      |    1            1
ERROR: LoadError: Some tests did not pass: 1 passed, 1 failed, 0 errored, 0 broken.

Answer 2

That is a loooong writeup containing a simple question in it. The answer is also simple: yes, adding an outer test set is the de-facto standard for achieving the goal.