Is it possible to have Common Test run test cases in a random order?

Question

I have a couple dozen Common Test test suite modules for an Erlang application I have written. All the tests are passing but I feel the test suites are very brittle. Reordering the tests causes some of them to fail. I didn't read the dependencies chapter in the Common Test documentation and I had often made assumption about the application state in the unit tests. Now I would like to make my test suite more robust.

Randomize Test Order?

Coming from Ruby, where Rspec runs tests in random order, I would love to have this same functionality in Common Test. Does anyone know if there is a way to randomize test order in Common Test? I haven't seen anything in the docs about randomizing the test order.

Randomize Return Values From all/0 and groups/0?

I also thought about altering the output of the all/0 and groups/0 callbacks. Right now they just return hardcoded lists. Perhaps I could randomize the ordering of the elements and have them run in different orders each time? Does anyone have any experience randomizing test order by changing the callback return values in Common Test? I would also need a way to rerun the tests in the order that caused them to fail like Rspec's --seed flag.

Thanks in advance!

Answer 1

Using shuffle or {shuffle, Seed} property in defining test groups can be helpful, this way:

groups() ->
    [{group1, [shuffle], [test1, test2, test3]},
     {group2, [shuffle], [test1, test2, test3]}].

If shuffle is specified, the cases in the group will be executed in random order. There are good examples in this official documentation.