Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to make sure TestNG runs methods on test classes in succession instead of interleaved?

Tags:

testng

The situation and the problem

I have several test classes, each with several test methods. All tests use the same test database in the background. Each test class initializes its database contents and then tests stuff in several test methods.

When I run each test individually, they all pass. But when I run several tests at the same time (either using maven or my IDE, IntelliJ), the methods of different test classes are run interleaved, eg. the database initialization of the second class runs after the first class has started but before all test methods of first class have been run so these methods will fail (because the database already contains second class's data).

Some things I've tried, and some more details

The simplest solution would be to force the TestNG runner to run the classes in succession (ie. wait for all the test methods of a test class to finish before running test methods of another class). Can this be done?

I can probably do this by specifying each class as a separate test in my suite, but I don't want to do this as this means I'd have to add something to the suite whenever I add a test class which is clumsy and error-prone.

Simply asking TestNG to not parallelize anything (eg. setting thread count to 1 or disabling parallel running) doesn't help here since methods still get run in the wrong order (though not simultaneously).

One option would be to use a different database for each test class, but I don't see a simple way to do this (using JPA and Guice).

I'm not currently using DBUnit, Unitils etc.; I don't know these tools very well but I got the impression the don't solve my problems.

I'm using JPA to initialize database in each test class (ie. create entity objects and presist them).

like image 232
mazi Avatar asked Jun 24 '13 09:06

mazi


1 Answers

Even in sequential mode TestNG could interleave test methods from the same suite. It does guarantee the sequence @BeforeClass -> @Test -> @AfterClass but it can do something like:

before class1
    test class1.method1
before class2
    test class2.method1
    test class1.method2
after class1
    test class2.method2
after class2

The solution is to force each class in a different suite (which are executed truly sequentially). As of version 2.16, the maven surefire plugin puts each class in a separate suite so the problem is fixed.

On the other hand, IDEA (even the latest 13 EAP) generates an xml file with all classes in the same suite. Hopefully IDEA will follow suit and fix this too. Interleaved tests are a showstopper when working with shared resources such as databases.

like image 167
Bogdan Calmac Avatar answered Sep 22 '22 00:09

Bogdan Calmac