Is Cucumber-jvm thread safe?

Question

I want to run the same Cucumber tests in multiple threads. More specifically, I have a set of features, and running these features in one thread works fine. I use the JSON formatter to record running time of each step. Now I want to do load test. I care more about the running time of each feature/step in a multi-thread environment. So I create multiple threads, and each thread runs on the same feature set. Each thread has its own JSON report. Is this possible in theory?

For some project setup reason I cannot use the JUnit runner. So I have to resort to the CLI-way:

        long threadId = Thread.currentThread().getId();
        String jsonFilename = String.format("json:run/cucumber%d.json", threadId);

            String argv[] = new String[]{
                "--glue",
                "com.some.package",
                "--format",
                jsonFilename,
                "d:\\features"};

        // Do not call Main.run() directly. It has a System.exit() call at the end.             
        // Main.run(argv, Thread.currentThread().getContextClassLoader());

        // Copied the same code from Main.run(). 
        ClassLoader classLoader = Thread.currentThread().getContextClassLoader();
        RuntimeOptions runtimeOptions = new RuntimeOptions(new Env("cucumber-jvm"), argv);
        ResourceLoader resourceLoader = new MultiLoader(classLoader);
        ClassFinder classFinder = new ResourceLoaderClassFinder(resourceLoader, classLoader);
        Runtime runtime = new Runtime(resourceLoader, classFinder, classLoader, runtimeOptions);
        runtime.writeStepdefsJson();
        runtime.run();      

I tried to create a seperate thread for each Cucumber run. The problem is, only one of the thread has a valid JSON report. All the other threads just create empty JSON files. Is this by design in Cucumber or is there something I missed?

Answer 1

We have looked into multi-threading cucumber tests under Gradle and Groovy using the excellent GPars library. We have 650 UI tests and counting.

We didn't encounter any obvious problems running cucumber-JVM in multiple threads but the multi-threading also didn't improve performance as much as we hoped.

We ran each feature file in a separate thread. There are a few details to take care of, like splicing together the cucumber reports from different threads and making sure our step code was thread-safe. We sometimes need to store values between steps, so we used a concurrentHashMap keyed to the thread id to store this kind of data:

class ThreadedStorage {
    static private ConcurrentHashMap multiThreadedStorage = [:]

    static private String threadSafeKey(unThreadSafeKey) {
        def threadId = Thread.currentThread().toString()
        "$threadId:$unThreadSafeKey"
    }

    static private void threadSafeStore(key, value) {
        multiThreadedStorage[threadSafeKey(key)] = value
    }

    def static private threadSafeRetrieve(key) {
        multiThreadedStorage[threadSafeKey(key)]
    }


}

And here's the gist of the Gradle task code that runs the tests multi-threaded using GPars:

def group = new DefaultPGroup(maxSimultaneousThreads())
def workUnits = features.collect { File featureFile ->
    group.task {
        try {
            javaexec {
                main = "cucumber.api.cli.Main"
                ...
                args = [
                     ...
                     '--plugin', "json:$unitReportDir/${featureFile.name}.json",
                             ...
                             '--glue', 'src/test/groovy/steps',
                             "path/to/$featureFile"
                    ]
            }
        } catch (ExecException e) {
                ++noOfErrors
                stackTraces << [featureFile, e.getStackTrace()]
        }
    }
}
// ensure all tests have run before reporting and finishing gradle task
workUnits*.join()

We found we needed to present the feature files in reverse order of execution time for best results.

The results were a 30% improvement on an i5 CPU, degrading above 4 simultaneous threads, which was a little disappointing.

I think the threads were too heavy for multi-threading on our hardware. Above a certain number of threads there were too many CPU cache misses.

Running concurrently on different instances using a thread-safe work queue like Amazon SQS now seems a good way forward, especially since it is not going to suffer from thread-safety issues (at least not on the test framework side).

It is non-trivial for us to test this multi-threading method on i7 hardware due to security constraints in our workplace, but I would be very interested to hear how an i7 with a larger CPU cache and more physical cores compares.

Answer 2

Not currently -- here is the issue you observe. I haven't found any way to parallelize by scenario.

Here's a nice write up on poor-man's concurrency. Just run multiple commands each selecting a different subset of your tests -- by feature or tag. I would fork a new JVM (as a JUnit driver would) rather than trying to thread it since cucumber was not designed for that. You have to balance them yourself, then figure out how to combine the reports. (But at least the problem is combining reports not corrupt reports.)