Unit test sequence when running all tests in solution

Question

In a complex VS2008 solution I have three unit test projects. As they operate on the same test database it is important that the test projects are executed one after the other. It is not important which project first, just that one is finished before the other starts.

If I want to execute them all, there are several ways to do that, which lead to different results:

• I have a test list .vsmdi file where the tests are ordered by project. If I open the list and execute the tests from the test list editor, everything is fine.
• If I open the Test View window, sort the tests by project and run them, again everything is fine.
• However if I run the tests by selecting 'Test -> Run -> All Tests in Solution' from the menu, they get executed in random order where some of them fail, as one of the other test projects already manipulated the test db.

So the question is, what determines the unit test sequence when using the third approach? Is there a way to specify a default test list in the .testrunconfig?

As there are workarounds, the issue is not critical at all. But any thoughts are welcome. Thanks.

Answer 1

You are experiencing a unit testing smell called Interactive Tests, described in the excellent xUnit Test Patterns. More specifically, you suffer from a Shared Fixture, which is another way of saying that several of your tests use the same shared database, and that they are dependent on the outcome of previous test runs.

This is an anti-pattern and should be avoided at all cost. The book offers extensive guidance on how to deal with this situation.

The reason I start out by describing this is that this is the reason that MSTest doesn't guarantee ordering of the tests. Although the order may seem (semi)deterministic, you can't rely on it. Some other unit testing frameworks (xUnit.NET comes to mind) even go so far as to run tests in a random order each time you run the test suite. MSTest isn't that extreme, but there are no way you can order your unit tests.

That said, if you are running Visual Studio Team Suite or (I believe) Team Test, there's a type of test called an Ordered Test. You can use that to specify that all tests (including unit tests) should execute in a specific order.