Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

how does MSTest determine the order in which to run test methods?

Tags:

edit: note, question 288805 is similar, however, I specifically am asking how does MSTest choose the default test order. Please see the rest of this question. Thank you Eilon for the link.

I was looking at a legacy MSTest project. The tests were always running in the same order. The order was not alphabetic and was bouncing between methods in two *.cs TestMethod files.

I did not change the physical order of the legacy code. I did for my convenience append "MSTest01" to the method name of the first test, "MSTest02" to the method name of the second test, et cetera.

To my surprise, the execution order of the TestMethod functions changed; #3 first, #6 second, #5 third, et cetera.

When I removed the "MSTestnn" strings from the TestMethod function names, their execution order changed back to the previous ordering, i.e., one test from the first .cs file, two tests from the second .cs file, five tests from the first .cs file, et cetera.

It seems that file location may not be a factor while TestMethod function name may be a factor.

QUESTION: can anyone explain how MSTest decides on execution order of TestMethod functions?

like image 328
gerryLowry Avatar asked Feb 12 '10 21:02

gerryLowry


People also ask

In what order the tests run in JUnit?

By default, JUnit runs tests using a deterministic but unpredictable order (MethodSorters. DEFAULT). In most cases, that behavior is perfectly fine and acceptable. But there are cases when we need to enforce a specific ordering.

What is the correct order of the unit test layout?

A typical unit test contains 3 phases: First, it initializes a small piece of an application it wants to test (also known as the system under test, or SUT), then it applies some stimulus to the system under test (usually by calling a method on it), and finally, it observes the resulting behavior.

Does MSTest run tests in parallel?

The biggest advantage of MSTest is that it allows parallelization at the method level. As opposed to the other frameworks which only allow parallelization at the class level. So, for example, If you have 100 test methods in 5 classes, MSTest will let you run 100 tests in parallel.


2 Answers

I believe that MSTest executes test methods ordering them by their 'ID' (seems to be their full namespace).

I created a TestProject1 wich contains 4 unt tests (UnitTest1, ...2, ...A, ...B). Each unit test contains 5 test methods (TestMethodA, ...B, ...1, ...2, ...3). They were declared with random order inside their test classes. Now, every time I run MSTest, the tests are executed with the same order:

TestProject1.UnitTest1.TestMethod1 TestProject1.UnitTest1.TestMethod2 TestProject1.UnitTest1.TestMethod3 TestProject1.UnitTest1.TestMethodA TestProject1.UnitTest1.TestMethodB TestProject1.UnitTest2.TestMethod1 TestProject1.UnitTest2.TestMethod2 TestProject1.UnitTest2.TestMethod3 TestProject1.UnitTest2.TestMethodA TestProject1.UnitTest2.TestMethodB TestProject1.UnitTestA.TestMethod1 TestProject1.UnitTestA.TestMethod2 TestProject1.UnitTestA.TestMethod3 TestProject1.UnitTestA.TestMethodA TestProject1.UnitTestA.TestMethodB TestProject1.UnitTestB.TestMethod1 TestProject1.UnitTestB.TestMethod2 TestProject1.UnitTestB.TestMethod3 TestProject1.UnitTestB.TestMethodA TestProject1.UnitTestB.TestMethodB 

The only way to change that order is to rename one TestClass or a TestMethod. If for example I rename the TestMethodB, of the UnitTest1, to TestMethod4 it will be executed before TestMethodA.

To see the IDs of your test methods open the 'Test View' window from VS and then right click on a column header (e.g. Test Name) --> "Add/Remove Columns..." and add 'ID' column.

like image 141
chaliasos Avatar answered Oct 03 '22 15:10

chaliasos


The MSDN says ;-)

How to: Create an Ordered Test

http://msdn.microsoft.com/en-us/library/ms182631.aspx

MSTest.exe command-line options

http://msdn.microsoft.com/en-us/library/ms182489(v=vs.120).aspx

like image 36
Kim Ki Won Avatar answered Oct 03 '22 14:10

Kim Ki Won