Unit Tests Fails in Build Service But Not Locally

Question

Two tests fail in my build service that will not fail when ran locally.

What I've discovered is that when I execute mvn clean test locally the sort order of my test classes are alphabetical (package and class), but when ran in the build service they're seemingly random.

It is not my intention of needing a specific "order" so I'm definitely concerned that one test prior to another is hinting at an isolation issue. However, I don't think I can just tell maven to run classes in a certain order.

What can I do to try to reproduce?

EDIT

I cloned my build plan, but pointed at a forked repo (same code) and it ran successfully with no failed tests...

Answer 1

You should try to execute mvn -Dsurefire.runOrder=random clean test for random order locally.

But even if your build fails locally, random order is not the best option to reproduce failures because of test order. If the tests are all green when run in alphabetical order, you might be able to make your build consistently fail with mvn -Dsurefire.runOrder=reversealphabetical clean test

There are also a couple of more options -- see runOrder documentation