Running “cabal test” passes although there are test failures

Question

I have a Cabal test target:

test-suite Tests
  type:              exitcode-stdio-1.0
  main-is:           Main.hs
  hs-source-dirs:    test, src
  build-depends:     base, …
  default-language:  Haskell2010

And a simple testing Main.hs:

import Test.HUnit
testSanity = TestCase $ assertEqual "Should fail" 2 1
main = runTestTT testSanity

Now running cabal test passes:

Test suite Tests: RUNNING...
Test suite Tests: PASS
Test suite logged to: dist/test/Project-0.1.0-Tests.log
1 of 1 test suites (1 of 1 test cases) passed.

Even though there are failures correctly logged in the test suite log:

Test suite Tests: RUNNING...

Cases: 1  Tried: 0  Errors: 0  Failures: 0

### Failure:
Should fail
expected: 2
 but got: 1
Cases: 1  Tried: 1  Errors: 0  Failures: 1
Test suite Tests: PASS
Test suite logged to: dist/test/Project-0.1.0-Tests.log

What am I doing wrong?

Answer 1

main has to be of type IO a where a can be anything, and that value is not tied in any way to the POSIX exit status of the program. You need to look at the Counts returned by runTest and explicitly choose a successful or failed output code.

import System.Exit

main :: IO ()
main = do
    cs@(Counts _ _ errs fails) <- runTestTT testSanity
    putStrLn (showCounts cs)
    if (errs > 0 || fails > 0) 
        then exitFailure
        else exitSuccess

Answer 2

I believe exitcode-stdio-1.0 expects the test to communicate failure via the program's exit code. However, if you run the test program manually from the shell (look in the dist/build directory) I think you'll find that the exit code is always 0.

Here is a related SO question which suggests that your test should call exitFailure if your suite doesn't pass:

QuickCheck exit status on failures, and cabal integration

FWIW, it works correctly when using stack:

$ mkdir new-dir
$ cd new-dir
$ stack new
(edit new-template.cabal to add HUnit as a test dependency)
(add test case to test/Spec.hs)
$ stack test

Perhaps stack is also scanning the test output.