How to handle parent test teardown with parallel subtests in golang

Question

Overview

If I have a parent test with setup and teardown logic, how do I run subtests within it in parallel without running into a race condition with the teardown logic?

func TestFoo(t *testing.T) {
    // setup logic
    t.Run("a", func(t *testing.T) {
        t.Parallel()
        // test code
    })
    // teardown logic
}

Example

As a contrived example: let's say that the test needs to create a tmp file that will be used by all subtests and delete it when the test is over.

For the example, the parent test also calls t.Parallel(), as that's what I ultimately want. But my problem and the output below is the same even if the parent does not call t.Parallel().

Sequential Subtest

If I run the subtests sequentially, they pass no problem:

package main

import (
    "fmt"
    "io/ioutil"
    "os"
    "testing"
)

func setup(t *testing.T) (tmpFile string) {
    f, err := ioutil.TempFile("/tmp", "subtests")
    if err != nil {
        t.Fatalf("could not setup tmp file: %+v", err)
    }
    f.Close()
    return f.Name()
}

var ncase = 2

func TestSeqSubtest(t *testing.T) {
    t.Parallel()

    // setup test variables
    fname := setup(t)

    // cleanup test variables
    defer func() {
        os.Remove(fname)
    }()

    for i := 0; i < ncase; i++ {
        t.Run(fmt.Sprintf("test_%d", i), func(t *testing.T) {
            if _, err := os.Stat(fname); os.IsNotExist(err) {
                t.Fatalf("file was removed before subtest finished")
            }
        })
    }
}  

Output:

$ go test subtests  
ok      subtests        0.001s

Parallel Subtest

If, however, I run the subtests in parallel, then the parent test's teardown logic ends up getting called before the subtest has a chance to run, making it impossible to get the subtest to run correctly.

This behavior, while unfortunate, fits with what the "Using Subtests and Sub-benchmarks" go blog says:

A test is called a parallel test if its test function calls the Parallel method on its instance of testing.T. A parallel test never runs concurrently with a sequential test and its execution is suspended until its calling test function, that of the parent test, has returned.

func TestParallelSubtest(t *testing.T) {
    t.Parallel()

    // setup test variables
    fname := setup(t)

    // cleanup test variables
    defer func() {
        os.Remove(fname)
    }()

    for i := 0; i < ncase; i++ {
        t.Run(fmt.Sprintf("test_%d", i), func(t *testing.T) {
            t.Parallel() // the change that breaks things
            if _, err := os.Stat(fname); os.IsNotExist(err) {
                t.Fatalf("file was removed before subtest finished")
            }
        })
    }
}

Output:

$ go test subtests  
--- FAIL: TestParallelSubtest (0.00s)
    --- FAIL: TestParallelSubtest/test_0 (0.00s)
        main_test.go:58: file was removed before subtest finished
    --- FAIL: TestParallelSubtest/test_1 (0.00s)
        main_test.go:58: file was removed before subtest finished
FAIL
FAIL    subtests        0.001s

Parallel Subtest with WaitGroup

As the above quote states, parallel subtests won't execute until their parent has finished, which means that trying to solve this with a sync.WaitGroup results in a deadlock:

func TestWaitGroupParallelSubtest(t *testing.T) {
    t.Parallel()
    var wg sync.WaitGroup

    // setup test variables
    fname := setup(t)

    // cleanup test variables
    defer func() {
        os.Remove(fname)
    }()

    for i := 0; i < ncase; i++ {
        t.Run(fmt.Sprintf("test_%d", i), func(t *testing.T) {
            wg.Add(1)
            defer wg.Done()
            t.Parallel()
            if _, err := os.Stat(fname); os.IsNotExist(err) {
                t.Fatalf("file was removed before subtest finished")
            }
        })
    }
    wg.Wait() // causes deadlock
}

output:

$ go test subtests  
--- FAIL: TestParallelSubtest (0.00s)
    --- FAIL: TestParallelSubtest/test_0 (0.00s)
        main_test.go:58: file was removed before subtest finished
    --- FAIL: TestParallelSubtest/test_1 (0.00s)
        main_test.go:58: file was removed before subtest finished
fatal error: all goroutines are asleep - deadlock!

goroutine 1 [chan receive]:
testing.tRunner.func1(0xc00009a000)
        /path/to/golang1.1.11/src/testing/testing.go:803 +0x1f3
testing.tRunner(0xc00009a000, 0xc00005fe08)
        /path/to/golang1.1.11/src/testing/testing.go:831 +0xc9
testing.runTests(0xc00000a0a0, 0x6211c0, 0x3, 0x3, 0x40b36f)
        /path/to/golang1.1.11/src/testing/testing.go:1117 +0x2aa
testing.(*M).Run(0xc000096000, 0x0)
        /path/to/golang1.1.11/src/testing/testing.go:1034 +0x165
main.main()
        _testmain.go:46 +0x13d

goroutine 7 [semacquire]:
sync.runtime_Semacquire(0xc0000a2008)
        /path/to/golang1.1.11/src/runtime/sema.go:56 +0x39
sync.(*WaitGroup).Wait(0xc0000a2000)
        /path/to/golang1.1.11/src/sync/waitgroup.go:130 +0x64
subtests.TestWaitGroupParallelSubtest(0xc00009a300)
        /path/to/go_code/src/subtests/main_test.go:91 +0x2b5
testing.tRunner(0xc00009a300, 0x540f38)
        /path/to/golang1.1.11/src/testing/testing.go:827 +0xbf
created by testing.(*T).Run
        /path/to/golang1.1.11/src/testing/testing.go:878 +0x353

goroutine 8 [chan receive]:
testing.runTests.func1.1(0xc00009a000)
        /path/to/golang1.1.11/src/testing/testing.go:1124 +0x3b
created by testing.runTests.func1
        /path/to/golang1.1.11/src/testing/testing.go:1124 +0xac

goroutine 17 [chan receive]:
testing.(*T).Parallel(0xc0000f6000)
        /path/to/golang1.1.11/src/testing/testing.go:732 +0x1fa
subtests.TestWaitGroupParallelSubtest.func2(0xc0000f6000)
        /path/to/go_code/src/subtests/main_test.go:85 +0x86
testing.tRunner(0xc0000f6000, 0xc0000d6000)
        /path/to/golang1.1.11/src/testing/testing.go:827 +0xbf
created by testing.(*T).Run
        /path/to/golang1.1.11/src/testing/testing.go:878 +0x353

goroutine 18 [chan receive]:
testing.(*T).Parallel(0xc0000f6100)
        /path/to/golang1.1.11/src/testing/testing.go:732 +0x1fa
subtests.TestWaitGroupParallelSubtest.func2(0xc0000f6100)
        /path/to/go_code/src/subtests/main_test.go:85 +0x86
testing.tRunner(0xc0000f6100, 0xc0000d6040)
        /path/to/golang1.1.11/src/testing/testing.go:827 +0xbf
created by testing.(*T).Run
        /path/to/golang1.1.11/src/testing/testing.go:878 +0x353
FAIL    subtests        0.003s

Summary

So how can I go about having a teardown method in the parent test that gets called after the parallel subtests run?

Answer 1

In the Go Blog on subtests it's mentioned how to do this:

func TestParallelSubtest(t *testing.T) {
    // setup test variables
    fname := setup(t)

    t.Run("group", func(t *testing.T) {
        for i := 0; i < ncase; i++ {
            t.Run(fmt.Sprintf("test_%d", i), func(t *testing.T) {
                t.Parallel()
                if _, err := os.Stat(fname); os.IsNotExist(err) {
                    t.Fatalf("file was removed before subtest finished")
                }
            })
        }
    })
    
    os.Remove(fname)
}

The relevant part of the blog post is under Control of Parallelism:

Each test is associated with a test function. A test is called a parallel test if its test function calls the Parallel method on its instance of testing.T. A parallel test never runs concurrently with a sequential test and its execution is suspended until its calling test function, that of the parent test, has returned. [...]

A test blocks until its test function returns and all of its subtests have completed. This means that the parallel tests that are run by a sequential test will complete before any other consecutive sequential test is run.

The specific solution for your problem can be found in the Cleaning up after a group of parallel tests section.