Run TestNG tests in random order

Question

Similarly to How can I make my JUnit tests run in random order? , I'd like TestNG to run my tests in random order, so unintended dependencies cannot creep in.

The TestNG manual states:

By default, TestNG will run the tests found in your testng.xml file in a random order.

However, I created a small test project, with a simple testng.xml:

<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="My suite">
    <test name="Simple test">
        <packages>
            <package name="testngtests"></package>
        </packages>
    </test>
</suite>

The package testngtests contains two test classes (MyTest1, MyTest2), and these contain a few empty methods like this:

@Test
public void testOne(){

}

The test mehods are all empty, and only differ by name.

When I run them (using the Eclipse TestNG runner, or on the command line), the tests consistenly run in the same order (namely sorted alphabetically, first by class and then by method name).

So is the documentation wrong?

Or does "in random order" simply mean "there is no guaranteed order"? Then how can I make TestNG actively randomize test order?

Answer 1

Yes, 'random' should really be 'non predictable'.

If you want true randomization, look up IMethodInterceptor, where TestNG will give you an opportunity to change the ordering to anything you like.