Create test framework for testing functions with different types

Question

I'm trying to create a testing framework for my homework assignment. The idea is to pass a list of tests (each of them consists of a function, parameters and and an expected value) to a function that will do the testing and then print a symbol that represents success, failure or an error (exception raised).

The problem is that functions as well as paramters may have different types like int * int -> int or string list -> bool. I store them in tuples but I can't put them in a list since they have different types.

One way to do this is to create a datatype that will have a constructor for each case but it's tedious. So my question is there a simple way to do this?

Answer 1

One trick is to package up all the functionality you need into a functions of type unit -> unit. Then you can put these into a list.

Let's build up a small example. I might implement a small testing framework like so:

datatype 'a result = Value of 'a | Raised of exn

type ('a, 'b) test =
  { func : 'a -> 'b
  , input : 'a
  , check : 'b result -> bool (* checks if the output is correct *)
  }

fun runTest {func, input, check} =
  let val result = Value (func input) handle e => Raised e
  in if check result then
       print "Correct\n"
     else
       case result of
         Value _ => print "Incorrect\n"
       | Raised e => print ("Raised " ^ exnMessage e ^ "\n")
  end

Here's an example usage:

val t1 =
  { func = op^
  , input = ("hello", "world")
  , check = (fn Value "helloworld" => true | _ => false)
  }
val t2 =
  { func = Word.fromInt
  , input = 5
  , check = (fn Value 0w5 => true | _ => false)
  }
val _ = runTest t1
val _ = runTest t2

But now, as you point out, you have the problem that you can't do this:

val tests = [t1, t2] (* doesn't typecheck *)
val _ = List.app runTest tests

To fix this, I'd recommend doing the following.

type test' = unit -> unit

fun make (t : ('a, 'b) test) : test' =
  (fn () => runTest t)

fun runTest' (t' : test') = t' ()

This approach packages up all the functionality you need from a test (that is, running it, checking that it is correct, etc.) into a function of type unit -> unit. Now, all of your "tests" are the same type, and can be put in a list.

val t1' = make t1
val t2' = make t2

val tests' = [t1', t2']
val _ = List.app runTest' tests

Answer 2

One way to do this is to create a datatype that will have a constructor for each case but it's tedious.

Yes, that is certainly tedious.

• Most simply you can just write tests as a set of value declarations:

  val test_fact_0 = fact 0 = 1
  val test_fact_1 = fact 5 = 120
  val test_fact_2 = fact 20 = 2432902008176640000
  val test_fact_3 = (fact ~1; false) handle Out_of_memory => true | _ => false
  val test_fact_4 = (fact 21; false) handle Overflow      => true | _ => false
  val test_fact_all = List.all (fn b => b) [
        test_fact_0, test_fact_1, test_fact_2, test_fact_3, test_fact_4 ]
  

You can also find several test frameworks for Standard ML from which you can draw inspiration.

• There are simple test frameworks like github.com/kvalle/sml-testing where you might write:

  test("fact 0 defined as base case 1", assert_equals_int(fact 0, 1));
  test("fact 5 is 120", assert_equals_int(fact 5, 120));
  test("fact 20 is the largest output for 64 bits",
       assert_equals_int(fact 20, 2432902008176640000));
  test("fact ~1 should run out of memory", assert_raises(fact, ~1, Out_of_memory));
  test("fact 21 should overflow", assert_raises(fact, 21, Overflow));
  

  An example of this framework's simplicity is that exception functions are expected to have exactly one argument. This limitation could be overcome by extending the framework or by wrapping your test function in a closure:

  test("foo", assert_raises(fn () => foo 1 2 3, (), Domain)
  

• You might be able to draw inspiration from github.com/br0ns/iptest, a test framework for an introductory programming course in SML. Unfortunately the combinators are in Danish, but a little search-and-replace could fix that. It uses a shallowly embedded domain-specific language to express the test suite. I.e., the following (translated) extract from eks2010/test.sml is valid SML code:

  examSet "Introduction to Programming 2010" is
  
    part "1" "Remove"
    note "Sub-part b cannot be tested automatically."
    test remove
      ? #"t" & explode "klatret" ==> (explode "klaret", 3)
      ? #"a" & [#"a"]            ==> ([], 0)
      ? #"c" & explode "abc"     ==> (explode "ab", 2)
    testEnd
  
  examEnd
  

  The general framework supplies the functions examSet, part, note, test, etc., as well as the higher-order operators ?, &, ==> used for composing a function that will eventually execute the testable function against the listed test-cases. And the specialised part supplies the function remove which isn't actually the tested function, but rather a stub that helps with pretty-printing:

  val fjern =
      funktion2
        "fjern" fjern $
        char & list char --> par (list char, int)
  

• And there are more industrial-looking frameworks like github.com/smlsharp/SMLUnit that uses the module system to provide a test module that corresponds to the module you're testing; e.g. Dictionary.sml is being tested in TestDictionary.sml:

  structure TestDictionary =
  struct
    open Dictionary
    structure Assert = SMLUnit.Assert
    structure Test = SMLUnit.Test
  
    fun testCreate0001 () =
        (create (); ())
  
    fun testExists0001 () =
        let val emptyDictionary = create ()
        in Assert.assertFalse (exists emptyDictionary 1); () end
  
    ...
  
    fun suite () =
        Test.labelTests
       [
          ("create0001", testCreate0001),
          ("exists0001", testExists0001),
          ...
        ]
  
  end;
  
  SMLUnit.TextUITestRunner.runTest
      {output = TextIO.stdOut}
      (TestDictionary.suite ());