How to use cabal with bash tests

Question

For my project, I've written some unit tests as bash scripts. There really was no reasonable way to write the tests in Haskell.

I'd like these scripts to run when I type cabal test. How do I make that happen?

Answer 1

This module will allow you to run all .sh scripts in a specific subdirectory as a test. Furthermore, this uses the test-framework package so that if you want, you can run the test as:

cabal test '--test-option=--jxml=dist/test/$test-suite.xml'

And you can then obtain junit-style XML from the tests. This is currently checked in in my project for testing cabal things. The test code:

import Data.List (isSuffixOf)
import Control.Applicative
import Test.Framework (defaultMain, testGroup, Test)
import Test.Framework.Providers.HUnit
import Test.HUnit (assertFailure)
import System.Directory
import System.Exit (ExitCode(..))
import System.Process


main :: IO ()
main = makeTests "test" >>= defaultMain

-- Make a test out of those things which end in ".sh" and are executable
-- Make a testgroup out of directories
makeTests :: FilePath -> IO [Test]
makeTests dir = do
  origDir <- getCurrentDirectory
  contents <- getDirectoryContents dir
  setCurrentDirectory dir
  retval <- mapM fileFunc contents
  setCurrentDirectory origDir
  return $ concat retval
  where
    fileFunc "." = return []
    fileFunc ".." = return []
    fileFunc f | ".sh" `isSuffixOf` f = do
      fullName <- canonicalizePath f
      isExecutable <- executable <$> getPermissions fullName
      let hunitTest = mkTest fullName
      return [testCase f hunitTest | isExecutable]
    fileFunc d = do
      fullName <- canonicalizePath d
      isSearchable <- searchable <$> getPermissions fullName
      if isSearchable
        then do subTests <- makeTests d
                return [testGroup d subTests]
        else return []
    mkTest fullName = do
      execResult <- system fullName
      case execResult of
        ExitSuccess -> return ()
        ExitFailure code -> assertFailure ("Failed with code " ++ show code)

I use this with this clause in my .cabal file:

test-suite BackflipShellTests
  type:                exitcode-stdio-1.0
  main-is:             BackflipShellTests.hs
  hs-source-dirs:      test
  build-depends:       backflip, base, test-framework-hunit,
                       test-framework, directory, process, HUnit
  default-language:    Haskell2010

Note that though I placed the .sh tests and the test module in the same directory (called test), there's no inherent reason to do so.

Answer 2

First you need to add a test-suite to your cabal file. For this you will use the exitcode-stdio test suite, which will look something like this:

Name:           foo
Version:        1.0
License:        BSD3
Cabal-Version:  >= 1.9.2
Build-Type:     Simple

Test-Suite test-foo
    type:       exitcode-stdio-1.0
    main-is:    test-foo.hs
    build-depends: base

The above example test-suite was taken from the Cabal documentation for test suites

Then in your test-foo.hs file you would run the bash script and die with an exception for a non-zero exit code. You can do this using System.Process:

-- test-foo.hs
import System.Exit (ExitSuccess)
import System.Process (system)

main = do
    -- This dies with a pattern match failure if the shell command fails
    ExitSuccess <- system "./myprog"
    return ()

Then you can run the above test using cabal test and it will report a test failure if your shell program has a non-zero exit code.