How to run a list of tests with VsTest.Console

Question

How can I run a list of tests using vstest.console.exe? The .vsmdi format offered a way to specify test lists, but that format is deprecated(?).

I can run an explicit list of tests on the command line, which essentially does exactly what I want, but if the number if tests is large (say a few hundred) then I will run out of command line space!

vstest.console MyTests.dll /Tests:Test1,Test2

Is there no way that I can trick vstest.console.exe into running a list of tests defined in any other way?

(edit: emphasis)

Note: I don't want to change the test code, e.g add test category attributes or change naming schemes so name matching will select the subset. I need it to run a list of tests.

The best I can think of is to run as many as I can fit within the max command line length, and repeat until the set is done, then merge. But if there is some way of loading a legacy vsmdi list or similar it would be a lot easier.

vstest.console MyTests.dll < testnames.txt

vstest.console MyTests.dll /Testlist:testnames.txt

Answer 1

You can list your tests in a text file of a particular format and then feed that into vstest.console.exe like so. Assume file is called mytests.orderedtest:

vstest.console mytests.orderedtest

The mytests.orderedtest has to be in a specific format. There's an easy way to create such a test from Visual Studio and then you can look at the contents.

First, in Visual Studio, right-click on the project in Solution Explorer, then select Add / Ordered Test. This will create an orderedtest file with a nice UI that you can add your tests. So, pick your tests from the list: Test1,Test2. That will create a file that looks something like this:

<?xml version="1.0" encoding="UTF-8"?>
    <OrderedTest name="mytests" storage="c:\src\MyTests\MyTests.orderedtest" id="ed4d22c5-ab9a-4ebd-954f-65ac4c034338" xmlns="http://microsoft.com/schemas/VisualStudio/TeamTest/2010">
  <TestLinks>
    <TestLink id="14c6766b-c22b-130a-ddb0-53d5ddd6eb1d" name="Test1" storage="..\bin\debug\MyTests.dll" type="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestElement, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
    <TestLink id="24c6766b-c22b-130a-ddb0-53d5ddd6eb1d" name="Test2" storage="..\bin\debug\MyTests.dll" type="Microsoft.VisualStudio.TestTools.TestTypes.Unit.UnitTestElement, Microsoft.VisualStudio.QualityTools.Tips.UnitTest.ObjectModel, Version=14.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a" />
  </TestLinks>
</OrderedTest>

If you are going to create this manually, outside of Visual Studio, note that the GUID in the id attribute is significant. It is the only way tests with the same name are differentiated across different fully-qualified class names. That is, the id is composed from namespace+class+method. This article explains it: http://blogs.msdn.com/b/aseemb/archive/2013/10/06/how-to-create-an-ordered-test-programmatically.aspx

Here's an routine that will convert the fully qualified method name into one of these GUIDs:

// convert the test (<Name space name>.<class name>.<test method name>) to a GUID
static Guid GuidFromString(string data) 
{
    SHA1CryptoServiceProvider provider = new SHA1CryptoServiceProvider();
    byte[] hash = provider.ComputeHash(System.Text.Encoding.Unicode.GetBytes(data));
    byte[] toGuid = new byte[16]; 
    Array.Copy(hash, toGuid, 16);
    return new Guid(toGuid); 
}