Haskell QuickCheck Test not running properly when run with Cabal

Question

Something did not make sense with a Cabal package I was developing, and I have boiled the issue down to the following example:

I have the following simple test module:

module Main where
import Test.QuickCheck (quickCheck)
main = quickCheck False

And the following Cabal file in the same directory:

name:                project
version:             0.1.0.0
cabal-version:       >= 1.10
build-type:          Simple

executable project
  main-is:             Main.hs
  build-depends:       base, QuickCheck
  default-language:    Haskell2010

test-suite project-test
  type:                exitcode-stdio-1.0
  main-is:             Main.hs
  build-depends:       base, QuickCheck
  default-language:    Haskell2010

The only other files in the directory are dist (created by Cabal upon build) and the cabal sandbox files. Both the executable and the test-suite refer to Main.hs, and therefore I would expect to get the same test results when running "cabal run" as when running "cabal test". Apparently, however, that is not the case.

"cabal run" gives:

Preprocessing executable 'project' for project-0.1.0.0...
Running project...
*** Failed! Falsifiable (after 1 test):

which makes sense because the property test 'quickCheck False' should fail. As expected this is the same result I get when running main in ghci.

"cabal test", however, gives:

Test suite project-test: RUNNING...
Test suite project-test: PASS
Test suite logged to: dist/test/project-0.1.0.0-project-test.log
1 of 1 test suites (1 of 1 test cases) passed.

Why the heck does "cabal test" pass the test case, whereas "cabal run" fails it as expected?

Answer 1

quickCheck doesn't exit the program, therefore, it doesn't set the exit code. After all, you could have several quickCheck tests, which should be independent of each other:

main = do
    quickCheck prop1 -- should this exit if the test fails?
    quickCheck prop2 -- should this exit if the test fails?
    quickCheck prop3 -- should this exit if the test fails?

However, you can easily fix this if you a) exit as soon as one of your tests doesn't pass or b) remember whether a single test hasn't passed and then exit with the correct code.

Using only QuickCheck, but no other library

Exit with failure as soon as a test fails

For this, you simply use quickCheckResult, which you can check with isSuccess from Test.QuickCheck.Test. If you want to use your current quickCheck definition, you can use qualified includes to exchange the default implementation with your special one:

import           Control.Monad          (when)    
import           System.Exit            (exitFailure)
import           Test.QuickCheck hiding (quickCheck, quickCheckWith)
import qualified Test.QuickCheck.Test as Q

quickCheckWith :: Testable prop => Args -> prop -> IO ()
quickCheckWith args p = do
      success <- fmap Q.isSuccess $ quickCheckWithResult args p 
      when (not success) $ exitFailure -- exit if the result is a failure

quickCheck :: Testable prop => prop -> IO ()
quickCheck p = quickCheckWith stdArgs p

You should probably use another name though, especially if other people work on the same project. checkOrExit would be plausible.

Return failure code but run all tests

This is essentially the same, but runs all test. You must use quickCheckResult or quickCheckWithResult again:

import           Control.Monad          (when)    
import           System.Exit            (exitFailure)
import           Test.QuickCheck        (quickCheckResult)
import           Test.QuickCheck.Test   (isSuccess)

main :: IO ()
main = do
  let tests = [ quickCheckResult prop1
              , quickCheckResult prop2
              , quickCheckResult prop3
              , quickCheckResult prop4
              ]
  success <- fmap (all isSuccess) . sequence $ tests
  when (not success) $ exitFailure

Using additional test libraries

While quickCheck is great for property checking, it doesn't provide a complete testing framework. That's where other full-fledged frameworks such as tasty or hspec come in handy. They can take a Testable a and inspect QuickCheck's result accordingly. An example using hspec:

module Main where
import Test.Hspec
import Test.QuickCheck (property)

main = hspec $ do
  describe "<method you would like to test>" $ 
    it "<property/assumption you would like to test>" $ 
       property $ False -- quickCheck

This gives the (more verbose) output

<method you would like to test>
  - <property/assumption you would like to test> FAILED [1]

1) <method you would like to test> <property/assumption you would like to test>
Falsifiable (after 1 test):
   [remark: the used values would be here, but `False` is a boolean]

Randomized with seed 2077617428

Finished in 0.0019 seconds
1 example, 1 failure

Also, it exits with an error code, so your cabal test will recognize this failed test correctly. Those test frameworks also have additional features, which go beyond of the scope of this answer.

TL;DR

QuickCheck doesn't exit your program, but Cabal only inspects the exit code to determine whether the tests have passed. Since any regularly ending Haskell program returns zero (aka no error), you either need to use exitFailure (or something similar), or a framework that uses exitFailure behind the scenes.

Note that this only holds for tests with type: exitcode-stdio-1.0.