Unit testing an agent

Question

I am trying to test a MailboxProcessor in F#. I want to test that the function f I am giving is actually executed when posting a message.

The original code is using Xunit, but I made an fsx of it that I can execute using fsharpi.

So far I am doing this :

open System 
open FSharp
open System.Threading
open System.Threading.Tasks



module MyModule =

    type Agent<'a> = MailboxProcessor<'a>
    let waitingFor timeOut (v:'a)= 
        let cts = new CancellationTokenSource(timeOut|> int)
        let tcs = new TaskCompletionSource<'a>()
        cts.Token.Register(fun (_) ->  tcs.SetCanceled()) |> ignore
        tcs ,Async.AwaitTask tcs.Task

    type MyProcessor<'a>(f:'a->unit) =
        let agent = Agent<'a>.Start(fun inbox -> 
             let rec loop() = async {

                let! msg = inbox.Receive()
                // some more complex should be used here
                f msg
                return! loop() 
             }
             loop()
        )

        member this.Post(msg:'a) = 
            agent.Post msg


open MyModule

let myTest =
    async {

        let (tcs,waitingFor) = waitingFor 5000 0

        let doThatWhenMessagepostedWithinAgent msg =
            tcs.SetResult(msg)

        let p = new MyProcessor<int>(doThatWhenMessagepostedWithinAgent)

        p.Post 3

        let! result = waitingFor

        return result

    }

myTest 
|> Async.RunSynchronously
|> System.Console.WriteLine 

//display 3 as expected

This code works, but it does not look fine to me.

1) is the usage of TaskCompletionSource normal in f# or is there some dedicated stuff to allow me waiting for a completion?

2) I am using a second argument in the waitingFor function in order to contraint it, I know I could use a type MyType<'a>() to do it, is there another option? I would rather not use a new MyType that I find cumbersome.

3) Is there any other option to test my agent than doing this? the only post I found so far about the subject is this blogpost from 2009 http://www.markhneedham.com/blog/2009/05/30/f-testing-asynchronous-calls-to-mailboxprocessor/

Answer 1

This is a tough one, I've been trying to tackle this for some time as well. This is what I found so far, it's too long for a comment but I'd hesitate to call it a full answer either...

From simplest to most complex, depends really how thoroughly you want to test, and how complex is the agent logic.

Your solution may be fine

What you have is fine for small agents whose only role is to serialize access to an async resource, with little or no internal state handling. If you provide the f as you do in your example, you can be pretty sure it will be called in a relatively short timeout of few hundred milliseconds. Sure, it seems clunky and it's double the size of code for all the wrappers and helpers, but those can be reused it you test more agents and/or more scenarios, so the cost gets amortized fairly quickly.

The problem I see with this is that it's not very useful if you also want to verify more than than the function was called - for example the internal agent state after calling it.

One note that's applicable to other parts of the response as well: I usually start agents with a cancellation token, it makes both production and testing life cycle easier.

Use Agent reply channels

Add AsyncReplyChannel<'reply> to the message type and post messages using PostAndAsyncReply instead of Post method on the Agent. It will change your agent to something like this:

type MyMessage<'a, 'b> = 'a * AsyncReplyChannel<'b>

type MyProcessor<'a, 'b>(f:'a->'b) =
    // Using the MyMessage type here to simplify the signature
    let agent = Agent<MyMessage<'a, 'b>>.Start(fun inbox -> 
         let rec loop() = async {
            let! msg, replyChannel = inbox.Receive()
            let! result = f msg
            // Sending the result back to the original poster
            replyChannel.Reply result
            return! loop()
         }
         loop()
    )

    // Notice the type change, may be handled differently, depends on you
    member this.Post(msg:'a): Async<'b> = 
        agent.PostAndAsyncReply(fun channel -> msg, channel)

This may seem like an artificial requirement for the agent "interface", but it's handy to simulate a method call and it's trivial to test - await the PostAndAsyncReply (with a timeout) and you can get rid of most of the test helper code.

Since you have a separate call to the provided function and replyChannel.Reply, the response can also reflect the agent state, not just the function result.

Black-box model-based testing

This is what I'll talk about in most detail as I think it's most general.

In case the agent encapsulates more complex behavior, I found it handy to skip testing individual messages and use model-based tests to verify whole sequences of operations against a model of expected external behavior. I'm using FsCheck.Experimental API for this:

In your case this would be doable, but wouldn't make much sense since there is no internal state to model. To give you an example what it looks like in my particular case, consider an agent which maintains client WebSocket connections for pushing messages to the clients. I can't share the whole code, but the interface looks like this

/// For simplicity, this adapts to the socket.Send method and makes it easy to mock
type MessageConsumer = ArraySegment<byte> -> Async<bool>

type Message =
    /// Send payload to client and expect a result of the operation
    | Send of ClientInfo * ArraySegment<byte> * AsyncReplyChannel<Result>
    /// Client connects, remember it for future Send operations
    | Subscribe of ClientInfo * MessageConsumer
    /// Client disconnects
    | Unsubscribe of ClientInfo

Internally the agent maintains a Map<ClientInfo, MessageConsumer>.

Now for testing this, I can model the external behavior in terms of informal specification like: "sending to a subscribed client may succeed or fail depending on the result of calling the MessageConsumer function" and "sending to an unsubscribed client shouldn't invoke any MessageConsumer". So I can define types for example like these to model the agent.

type ConsumerType =
    | SucceedingConsumer
    | FailingConsumer
    | ExceptionThrowingConsumer

type SubscriptionState =
    | Subscribed of ConsumerType
    | Unsubscribed

type AgentModel = Map<ClientInfo, SubscriptionState>

And then use FsCheck.Experimental to define the operations of adding and removing clients with differently successful consumers and trying to send data to them. FsCheck then generates random sequences of operations and verifies the agent implementation against the model between each steps.

This does require some additional "test only" code and has a significant mental overhead at the beginning, but lets you test relatively complex stateful logic. What I particularly like about this is that it helps me test the whole contract, not just individual functions/methods/messages, the same way that property-based/generative testing helps test with more than just a single value.

Use Actors

I haven't gone that far yet, but what I've also heard as an alternative is using for example Akka.NET for full-fledged actor model support, and use its testing facilities which let you run agents in special test contexts, verify expected messages and so on. As I said, I don't have first-hand experience, but seems like a viable option for more complex stateful logic (even on a single machine, not in a distributed multi-node actor system).