What's the best way to unit test concurrent Erlang code?

Question

I'm spending a bit of time with Erlang, and I'm wanting to apply TDD to code I'm writing.

While EUnit in the standard lib provides a nice traditional unit testing framework for testing regular style code, there doesn't seem to be anything to help specifically with testing concurrent code, which is used a LOT in Erlang.

Note that we're talking Erlang here, which uses message passing (as opposed to shared state) for communication between concurrent processes, so techniques for unit testing concurrent code in shared state languages may not be applicable.

Anyone found a good way to test concurrent code in Erlang?

Answer 1

I just found some cool newly (as of 2011) developed software for testing concurrent Erlang applications called Concuerror. There is a few papers on it and a repository on github. Apparently it works by using its own scheduler and systematically testing different interleaving between the processes.

Also worth mentioning is Dialyzer (DIscrepancy AnaLYZer for ERlang) (Papers, Tutorial, Manual) , which is a tool for static analysis of the code for finding errors. This has support for detecting some concurrency-errors too (see paper).

I have not tested any of these myself, though dialyzer seems to be relatively mature software. Both programs have a GUI for working with the tests.

PS. For the non-concurrent parts, EUnit and QuickCheck (there is a free version also) should work fine.

Answer 2

The question is a bit vague ("Erlang is concurrent, test it with Erlang!") but I'll try to elaborate a bit.

Testing Erlang code can range from the straightforwardly simple (the right input produces the right output) to setting up complex test harnesses that verify your component behaves the way it should. What is best for your given situation depends totally on the requirements you have and the amount of black box / white box testing you want to do.

Part of the beauty of Erlang is the ability to make concurrency transparent. Consider the following example (a function that parallelizes the summation of a list of lists):

deep_sum(ListOfLists) ->
     Parent = self(),
     [spawn(fun() -> Parent ! lists:sum(List) end) || List <- ListOfLists],
     lists:sum([receive Sum -> Sum end || _ <- ListOfLists]).

You would typically test this with a very simple EUnit test case:

deep_sum_test() ->
     ?assertEqual(0,  deep_sum([0,  0,  0,  0])),
     ?assertEqual(40, deep_sum([10, 10, 10, 10]).

Now, let's say we have a bit more explicit API to this functionality: a process pool as argument:

deep_sum(Pool, ListOfLists) ->
     distribute_lists(Pool, ListOfLists),
     lists:sum([receive Sum -> Sum end || _ <- ListOfLists]).

distribute_lists(Pool, ListOfLists) -> distribute_lists(Pool, Pool, ListOfLists).

distribute_lists([P|Pool], All, [L|ListOfLists]) ->
     P ! {self(), L},
     distribute_lists(Pool, All, ListOfLists);
distribute_lists([], All, ListOfLists) ->
     distribute_lists(All, All, ListOfLists);
distribute_lists(_Pool, _All, []) ->
     ok.

When testing this, we have to deal with faking this process pool:

deep_sum_test() ->
     Pool = [spawn_link(fun() -> fake_pool(1) end) || _ <- lists:seq(1, 3)],
     ?assertEqual(4, deep_sum(Pool, [lists:seq(1, 3) || _ <- list:seq(1, 4)]),
     ?assertEqual(7, deep_sum(Pool, [lists:seq(1, 3) || _ <- list:seq(1, 7)]),
     [P ! stop || P <- Pool].

fake_pool(CannedResponse) ->
     receive
         {From, _L} -> From ! CannedResponse;
         stop -> ok
     end,
     fake_pool(CannedResponse).

As you can see, testing concurrency programs in Erlang can take different shapes. These are extremely simple examples, but with Erlang's built in concurrency primitives it is very easy to create the kind of test harness you want, abstracting at the right levels.

I usually find TDD to be orthogonal to wether you're testing concurrent code or not, so said testing techniques can be used for normal unit testing as well.