Unit Tests from multiple test classes appear to be interleaved

Question

I have tests in two separate classes within a MSTest project, each class is (I think) set up properly and each class runs fine but when I run the whole project's tests, some fail.

My two classes both involve setting up some external data but each class is designed to make sure this is in a known state before starting (and to wipe it after finishing). I would have though MSTest might run all methods in a test class in parallel(?), but would run each class in sequence... is this an incorrect assumption?

Edit: I came across this question (how does MSTest determine the order in which to run test methods?) It seems to suggest VS may interleave tests from multiple classes in a (seemingly but not actually) random order i.e. it does not run all tests in classA before starting in classB. This is problematic in my case because even if I order my tests within a class, MSTest might still run methods from multiple classes which conflict?

Answer 1

Important note

MsTest doesn't guarantee the order of execution and order can be different between runs. If you need to rely on order you'll need to created an Ordered Test, a VS Premium/Enterprise feature unless you're on Visual Studio 2015 update 2, which brought them to Pro.

Test execution can interleave, bounce and do all kinds of crazy things. Though on some versions of MsTest it tends to run in alphabetical order of the full namespace of the test, e.g.: my.namespace.class.method.

When using data driven tests it's even weirder as the order of the data coming from the datasource acts as an additional randomizer.

Do not rely on order It's probably better to use TestInitialize and run your code before each test executes than to rely on [Class|Assembly]Initialize to setup your data in ways that are incompatible between different tests.

Visual Studio 2015 and earlier.

MsTest can execute tests in parallel using the legacy test runner and when specifying parallelism in the testSettings file. In the legacy runner tests will be executed on up to 5 threads. The legacy test runner will not constrain the tests in any way. It will run all tests in parallel if it can and there is no specific order or protection. You can see in the screenshot below:

Unless your testSettings explicitly mention the parallism, the tests will run sequentially, but with no specific order. I suspect it will use the IL order of the compiled assembly or alphabetical order. The order in the source file is definitely not what is used.

In Visual Studio 2015 update 1 and later

Visual Studio 2015 update 1 introduced the option to run tests in parallel in the new test runner. The new test runner uses a different approach and will parallellize the tests per container. For C# this means this means that tests in one assembly are executed sequentially while tests that are in separate assemblies can be executed in parallel.

It is assumed that Integration tests and UI tests are kept in their own separate solution and that when you enable this parallel option you have a solution containing only Unit tests.