Run/Exclude Specific Tests in Build Definition Team Foundation Server 2015

Question

In one of our build definitions I'm trying to configure a 'Visual Studio Test' build step to run a specific set of tests.

There is a 'Test Filter critera' box (below) but this doesn't seem to have any effect, running all tests regardless. It's not like the search filter box in Test Explorer in VS2015, and there are some references to a 'TestCaseFilter' but this doesn't apply here (and might be TFS2012 only).

Can this be used to actually filter tests, or should I be doing it some other way (e.g. provide a settings file?)

FWIW we're using Xunit for our unit tests.

Answer 1

If the goal is to exclude a test of tests from a run that can be accomplished via

TestCategory!={name}

In my case I typically use Integration or Manual to indicate a non-CI test so my filter is

TestCategory!=Integration&TestCategory!=Manual

Answer 2

The Test Filter Criteria field is the best way to filter tests. You do have to add an attribute to your tests to categorize your tests. Here is post that explains this feature:

https://dotnetcatch.com/2016/03/11/vststfs-visual-studio-test-task-filter-criteria/

Answer 3

Test Filter Criteria filters tests from within the test assembly files. This option works the same way as the console option /TestCaseFilter of vstest.console.exe, you can test with vstest.console.exe command line to see whether it works.

As an alternate, you can specify the tests in Test Assembly. This field specifies the test assemblies(s) from which the tests should be picked.

• Wildcards can be used
• Multiple paths can be specified separated by a semicolon
• Paths are relative to the Sources Directory

Answer 4

For those (like me) with a more specific issue, where all the tests of a certain project have to be excluded, use the following statement in the "Test Assemblies" field:

!**\*[ProjectName]*.dll

In my case, I have one project that contains a lot of UI tests which are set up as unit tests. Initially the agent would try to run these tests as well, while they should only run at a specific release. An example of the fix:

This makes sure all the tests within that project are ignored, even though the dll contains "tests" in the name.