How to do test-driven development in ocaml?

Question

I think everything is in the title but I am looking specifically for:

• What is the "standard" unit test framework in Ocaml?
• How do I integrate executing tests in the build?
• How to automatically execute tests upon every file change?

As a bonus, I would be interested in test coverage tooling...

Answer 1

It seems that the package ounit enjoys quite a large popularity, there are several other packages like kaputt or broken – I am the author of the latter.

I guess you are interested as the specific part of TDD where tests can be automated, here is how I do it on my own projects. You can find a few examples on GitHub such as Lemonade or Rashell that both have a test suite found in their respective testsuite folders.

Usually I work according to the according workflow:

1. I start to work simultaneously on tests and interface (.mli) files, this way I write a minimal program and do not only write a test case for the functions I want to implement but also have the opportunity to experiment with interfaces to be sure that I have an easy-to-use interface.

For instance, for the interface to the find(1) command found in Rashell_Posix I started by writing test cases:

open Broken
open Rashell_Broken
open Rashell_Posix
open Lwt.Infix

let spec base = [
  (true,  0o700, [ base; "a"]);
  (true,  0o750, [ base; "a"; "b"]);
  (false, 0o600, [ base; "a"; "b"; "x"]);
  (false, 0o640, [ base; "a"; "y" ]);
  (true,  0o700, [ base; "c"]);
  (false, 0o200, [ base; "c"; "z"]);
]

let find_fixture =
  let filename = ref "" in
  let cwd = Unix.getcwd () in
  let changeto base =
    filename := base;
    Unix.chdir base;
    Lwt.return base
  in
  let populate base =
    Toolbox.populate (spec base)
  in
  make_fixture
    (fun () ->
       Lwt_main.run
         (Rashell_Mktemp.mktemp ~directory:true ()
          >>= changeto
          >>= populate))
    (fun () ->
       Lwt_main.run
         (Unix.chdir cwd;
          rm ~force:true ~recursive:true [ !filename ]
          |> Lwt_stream.junk_while (fun _ -> true)))

let assert_find id ?expected_failure ?workdir predicate lst =
  assert_equal id ?expected_failure
    ~printer:(fun fft lst -> List.iter (fun x -> Format.fprintf fft " %S" x) lst)
    (fun () -> Lwt_main.run(
         find predicate [ "." ]
         |> Lwt_stream.to_list
         |> Lwt.map (List.filter ((<>) "."))
         |> Lwt.map (List.sort Pervasives.compare)))
    ()
    lst

The spec and find_fixture functions are used to create a file hierarchy with the given names and permissions, to exercise the find function. Then the assert_find function prepares a test-case comparing the results of a call to find(1) with the expected results:

  let find_suite =
    make_suite ~fixture:find_fixture "find" "Test suite for find(1)"
    |& assert_find "regular" (Has_kind(S_REG)) [
      "./a/b/x";
      "./a/y";
      "./c/z";
    ]
    |& assert_find "directory" (Has_kind(S_DIR)) [
      "./a";
      "./a/b";
      "./c"
    ]
    |& assert_find "group_can_read" (Has_at_least_permission(0o040)) [
      "./a/b";
      "./a/y"
    ]
    |& assert_find "exact_permission" (Has_exact_permission(0o640)) [
      "./a/y";
    ]

Simultaneously I was writing on the interface file:

(** The type of file types. *)
type file_kind = Unix.file_kind =
  | S_REG
  | S_DIR
  | S_CHR
  | S_BLK
  | S_LNK
  | S_FIFO
  | S_SOCK

(** File permissions. *)
type file_perm = Unix.file_perm

(** File status *)
type stats = Unix.stats = {
  st_dev: int;
  st_ino: int;
  st_kind: file_kind;
  st_perm: file_perm;
  st_nlink: int;
  st_uid: int;
  st_gid: int;
  st_rdev: int;
  st_size: int;
  st_atime: float;
  st_mtime: float;
  st_ctime: float;
}

type predicate =
  | Prune
  | Has_kind of file_kind
  | Has_suffix of string
  | Is_owned_by_user of int
  | Is_owned_by_group of int
  | Is_newer_than of string
  | Has_exact_permission of int
  | Has_at_least_permission of int
  | Name of string (* Globbing pattern on basename *)
  | And of predicate list
  | Or of predicate list
  | Not of predicate

val find :
  ?workdir:string ->
  ?env:string array ->
  ?follow:bool ->
  ?depthfirst:bool ->
  ?onefilesystem:bool ->
  predicate -> string list -> string Lwt_stream.t
(** [find predicate pathlst] wrapper of the
    {{:http://pubs.opengroup.org/onlinepubs/9699919799/utilities/find.html} find(1)}
    command. *)

2. Once I was pleased with my test-cases and interfaces, I could try to compile them, even without an implementation. This is possible with bsdowl by just giving an interface file instead of an implementation file in the Makefile. Here compilation probably uncovered a few type errors in my tests that I could fix.

3. When the test compiled against the interface, I could implement the function, starting with an alibi function:

   let find _ = failwith "Rashell_Posix.find: Not implemented"

With this implementation I was able to compile my library and my test-suite. Of-course at this point, the test just fails.

4. At that point, I just needed to implement the Rashell_Posix.find function and iterate the tests until they passed.

This is how I do test-driven development in OCaml when I use automated tests. Some persons see interacting with the REPL as a form of test-driven development, this is a technique that I also like to use, it is rather straightforward to setup and use. The only setup step to use this latter form of test-driven-development in Rashell was to write an .ocamlinit file for the toplevel loading all the required libraries. This file looks like:

#use "topfind";;
#require "broken";;
#require "lemonade";;
#require "lwt.unix";;
#require "atdgen";;
#directory "/Users/michael/Workshop/rashell/src";;
#directory "/Users/michael/obj/Workshop/rashell/src";;

The two #directory directives correspond to the directories for sources and objects.

(Disclaimer: if you look carefully at the history, you will find that I took some liberties with the chronology, but there are other projects where I proceed exactly this way – I just cannot remember precisely which ones.)