How to test concurrency and locking in golang?

Question

We are trying to test locks. Basically, there are multiple clients trying to obtain a lock on a particular key. In the example below, we used the key "x".

I don't know how to test whether the locking is working. I can only read the logs to determine whether it is working.

The correct sequence of events should be:

1. client1 obtains lock on key "x"
2. client2 tries to obtain lock on key "x" (fmt.Println("2 getting lock")) - but is blocked and waits
3. client1 releases lock on key "x"
4. client2 obtains lock on key "x"

Q1: How could I automate the process and turn this into a test?

Q2: What are some of the tips to testing concurrency / mutex locking in general?

func TestLockUnlock(t *testing.T) {
    client1, err := NewClient()
    if err != nil {
        t.Error("Unexpected new client error: ", err)
    }

    fmt.Println("1 getting lock")
    id1, err := client1.Lock("x", 10*time.Second)
    if err != nil {
        t.Error("Unexpected lock error: ", err)
    }
    fmt.Println("1 got lock")

    go func() {
        client2, err := NewClient()
        if err != nil {
            t.Error("Unexpected new client error: ", err)
        }
        fmt.Println("2 getting lock")
        id2, err := client2.Lock("x", 10*time.Second)
        if err != nil {
            t.Error("Unexpected lock error: ", err)
        }
        fmt.Println("2 got lock")

        fmt.Println("2 releasing lock")
        err = client2.Unlock("x", id2)
        if err != nil {
            t.Error("Unexpected Unlock error: ", err)
        }
        fmt.Println("2 released lock")
        err = client2.Close()
        if err != nil {
            t.Error("Unexpected connection close error: ", err)
        }
    }()

    fmt.Println("sleeping")
    time.Sleep(2 * time.Second)
    fmt.Println("finished sleeping")

    fmt.Println("1 releasing lock")
    err = client1.Unlock("x", id1)
    if err != nil {
        t.Error("Unexpected Unlock error: ", err)
    }

    fmt.Println("1 released lock")

    err = client1.Close()
    if err != nil {
        t.Error("Unexpected connection close error: ", err)
    }

    time.Sleep(5 * time.Second)
}

func NewClient() *Client {
   ....
}

func (c *Client) Lock(lockKey string, timeout time.Duration) (lockId int64, err error){
   ....
}

func (c *Client) Unlock(lockKey string) err error {
   ....
}

Answer 1

Concurrency testing of lock-based code is hard, to the extent that provable-correct solutions are difficult to come by. Ad-hoc manual testing via print statements is not ideal.

There are four dynamic concurrency problems that are essentially untestable (more). Along with the testing of performance, a statistical approach is the best you can achieve via test code (e.g. establishing that the 90 percentile performance is better than 10ms or that deadlock is less than 1% likely).

This is one of the reasons that the Communicating Sequential Process (CSP) approach provided by Go is better to use than locks on share memory. Consider that your Goroutine under test provides a unit with specified behaviour. This can be tested against other Goroutines that provide the necessary test inputs via channels and monitor result outputs via channels.

With CSP, using Goroutines without any shared memory (and without any inadvertently shared memory via pointers) will guarantee that race conditions don't occur in any data accesses. Using certain proven design patterns (e.g. by Welch, Justo and WIllcock) can establish that there won't be deadlock between Goroutines. It then remains to establish that the functional behaviour is correct, for which the Goroutine test-harness mentioned above will do nicely.