What is the idiomatic testing strategy for GenServers in Elixir?

Question

I am writing a module to query an online weather API. I decided to implement it as an Application with a supervised GenServer.

Here is the code:

defmodule Weather do
  use GenServer

  def start_link() do
    GenServer.start_link(__MODULE__, :ok, name: __MODULE__)
  end

  def weather_in(city, country) do
    GenServer.call(__MODULE__, {:weather_in, city, country_code})
  end

  def handle_call({:weather_in, city, country}) do
    # response = call remote api
    {:reply, response, nil}
  end
end

In my test I decided to use a setup callback to start the server:

defmodule WeatherTest do
  use ExUnit.Case

  setup do
    {:ok, genserver_pid} = Weather.start_link
    {:ok, process: genserver_pid}
  end

  test "something" do
    # assert something using Weather.weather_in
  end

  test "something else" do
    # assert something else using Weather.weather_in
  end
end

I decided to register the GenServer with a specific name for several reasons:

• it is unlikely that someone would need multiple instances

• I can define a public API in my Weather module that abstracts the existence of an underlying GenServer. Users won't have to provide a PID/Name to the weather_in function to communicate with the underlying GenServer

• I can place my GenServer under a supervision tree

When I run the tests, as they run concurrently, the setup callback is executed once per test. Therefore there are concurrent attempts to start my server and it fails with {:error, {:already_started, #PID<0.133.0>}}.

I asked on Slack if there is anything I can do about it. Perhaps there is an idiomatic solution that I am not aware of...

To summarise the solutions discussed, when implementing and testing a GenServer, I have the following options:

1. Not registering the server with a specific name to let each test start its own instance of the GenServer. Users of the server can start it manually but they must provide it to the public API of the module. The server can also be placed in a supervision tree, even with a name but the public API of the module will still need to know which PID to talk to. Given a name passed as a parameter, I guess they could find the associated PID (I suppose OTP can do that.)

2. Registering the server with a specific name (like I did in my samples). Now there can be only one GenServer instance, tests must run sequentially (async: false) and each test must start and terminate the server.

3. Registering the server with a specific name. Tests can run concurrently if they all run against the same unique server instance (Using setup_all, an instance can be started only once for the whole test case). Yet, imho this is a wrong approach to testing as all tests will run against the same server, changing its state and therefore messing with each other.

Considering the users may not need to create several instances of this GenServer, I'm tempted to trade the tests concurrency for simplicity and go with solution 2.

[Edit] Trying solution 2 but it still fails for the same reason :already_started. I read again the docs about async: false and found out that it prevents the test case from running in parallel with other test cases. It doesn't run the tests of my test case in sequence as I thought. Help!

Answer 1

One crucial problem I note is that you have the wrong signature for handle_call, which should be handle_call(args, from, state) (you currently have just handle_call(args).

I've never used it, but those I look up to swear that QuickCheck is the gold standard for really testing GenServers.

At the unit test level, another option exists because of the functional architecture of GenServer:

If you test the handle_[call|cast|info] methods with expected argument and state combinations, you do NOT* have to start the GenServer: use your testing library to replace OTP, and call out to your module code as if it were a flat library. This won't test your api function calls, but if you keep those as thin pass-thru methods, you can minimize the risk.

*if you are using delayed replies, you'll have some problems with this approach, but you can probably sort them out with enough work.

I've made a couple changes to your GenServer:

• Your module doesn't use it's state, so I've made it more interesting from a testing perspective by adding an alternative premium webservice.
• I corrected the handle_call signature
• I added an internal State module to track state. Even on my GenServers without state, I always create this module for later, when I inevitably add state in.

The new module:

defmodule Weather do
  use GenServer

  def start_link() do
    GenServer.start_link(__MODULE__, [], name: __MODULE__)
  end

  def weather_in(city, country) do
    GenServer.call(__MODULE__, {:weather_in, city, country_code})
  end

  def upgrade, do: GenServer.cast(__MODULE__, :upgrade)

  def downgrade, do: GenServer.cast(__MODULE__, :downgrade)

  defmodule State do
    defstruct url: :regular
  end

  def init([]), do: {:ok, %State{}}

  def handle_cast(:upgrade, state) do
    {:noreply, %{state|url: :premium}}
  end
  def handle_cast(:downgrade, state) do
    {:noreply, %{state|url: :regular}}
  end

  # Note the proper signature for handle call:
  def handle_call({:weather_in, city, country}, _from, state) do
    response = case state.url do
      :regular ->
        #call remote api
      :premium ->
        #call premium api
    {:reply, response, state}
  end
end

and the testing code:

# assumes you can mock away your actual remote api calls
defmodule WeatherStaticTest do
  use ExUnit.Case, async: true

  #these tests can run simultaneously
  test "upgrade changes state to premium" do
    {:noreply, new_state} = Weather.handle_cast(:upgrade, %Weather.State{url: :regular})
    assert new_state.url == :premium
  end
  test "upgrade works even when we are already premium" do
    {:noreply, new_state} = Weather.handle_cast(:upgrade, %Weather.State{url: :premium})
    assert new_state.url == :premium
  end
  # etc, etc, etc...
  # Probably something similar here for downgrade

  test "weather_in using regular" do
    state = %Weather.State{url: :regular}
    {:reply, response, newstate} = Weather.handle_call({:weather_in, "dallas", "US"}, nil, state)
    assert newstate == state   # we aren't expecting changes
    assert response == "sunny and hot"
  end
  test "weather_in using premium" do
    state = %Weather.State{url: :premium}
    {:reply, response, newstate} = Weather.handle_call({:weather_in, "dallas", "US"}, nil, state)
    assert newstate == state   # we aren't expecting changes
    assert response == "95F, 30% humidity, sunny and hot"
  end
  # etc, etc, etc...      
end

Answer 2

I apologize for just now noticing this question and response so late in the process. I do believe the response given is of high quality. That said, I need to make a couple of points that can help you when doing testing harnesses. First note from the ExUnit.Callbacks documentation that

The setup_all callbacks are invoked once to setup the test 
case before any test is run and all setup callbacks are run 
before each test. No callback runs if the test case has no tests 
or all tests were filtered out.

Without review of the underlying code, this appears to imply that using the setup do/end block in the test file is tantamount to executing that bit of code before each test. It is handy to only have to write it once.

Now on to a different method entirely, I will use "doctests" in the code to define both the code and the tests. Similar to python doctests we can include test cases in the module documentation. These tests are executed with "mix test" as per norm. However, the tests live within the documentation and have the drawback of explicitly starting the server every time (as opposed to the implicit method of setup/do/end in the separate test file case.

From the documentation you will see that a document test can be initiated in a document block by indenting four spaces and putting iex> command.

I like the work by @chris meyer. Here I will take his work and do somethig a little different. I will actually test the api functions instead of the handle functions. It is a matter of taste and style and I have done exactly what Chris has done many times. I just think it is instructive to see the doctest form as it is also quite common and in the cases of a complex API function that is nt a simple pass through it is valuable to test the API function itself. So, using a snippet of Chris's, here is what I would do.

@doc """
Start our server.

### Example

We assert that start link gives :ok, pid

    iex> Weather.start_link
    {:ok, pid}
"""
def start_link() do
  GenServer.start_link(__MODULE__, [], name: __MODULE__)
end

@doc """
We get the weather with this funciton.

    iex> {:ok, pid} = Weather.start_link
    iex> Weather.in(pid, "some_city", "country_code")
    expected_response
    iex> Weather.in(pid, "some_other_city", "some_other_code")
    different_expected_response
"""
def weather_in(svr, city, country) doc
  GenServer.call(svr, {:weather_in, city, country_code})
end

The above technique has a couple of advantages:

1. The Elixir documentation is automatically generated when compiled
2. The documentation passes the "mix test" command, so what they see in your docs you KNOW works
3. Your mix symantics are the same with "mix test"

I had a little trouble with formatting with the code editor, so if someone wants to edit this a bit, please do so.