Yaml: How to exclude some tests in VSTest@2

Question

Currently, we're moving to AzureDevOps. We have some unit tests that fails on the agent but we don't have the capacity to fix them immediately. Until we fix them we'd like to exclude them from the pipeline and set a policy that all the test should pass, then fix those one-by-one.

Let me say, I have 4 exes, namely ABE_uTest.exe, BBE_uTest.exe, CBE_uTest.exe and DBE_uTest.exe. I'd like to exclude BBE and CBE. On the documentation page (AzureDevOps VsTest) I've found how to exclude files.

testAssemblyVer2: | # Required when testSelector == TestAssemblies
# *test.dll
# !*TestAdapter.dll
# !\obj*

So I assume simply a *_uTest.exe !BBE* !CBE* would work. But I miss something, because I can't make it work. Either it filters nothing, or everything. Here's the corresponding part of the yaml.

- task: VSTest@2
  displayName: VsTest - testAssemblies
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: '*_uTest.exe
      !*BBE*'
    searchFolder: '$(BuildPlatform)_$(BuildConfiguration)\testbin\'

The testAssemblyVer2 would be the setting. I've tried with only one exclude binary to make it more simple. I tried with and without ' characters at the beginning and the end. Also tried in one line and in separate lines but with no success. I always got this message:

##[warning]No test sources found matching the given filter '*_uTest.exe !BBE_uTest.exe'

What is the correct syntax to exclude files in AzureDevOps using yaml?

Answer 1

The notation is quite specific, you are missing a | and need to place it on several lines (other notations might exist, this works). Added a **\ for searching in every possible folder:

- task: VSTest@2
inputs:
  testSelector: 'testAssemblies'
  # note the |
  testAssemblyVer2: |
    **\*_uTest.exe
    !**\*BBE*

Why the |? It signals that you want to input several different items. 'Something something2' means look for a very long name with a space in it. Without the | several lines are seen as 1 input. See the documentation.

Answer 2

I was able to skip the tests using the testFiltercriteria input property of VsTest task, as shown below:

- task: VSTest@2
  inputs:
    testSelector: 'testAssemblies'
    testAssemblyVer2: |
      **\*_AlluTests.dll
    testFiltercriteria: 'FullyQualifiedName!~TestCase1&FullyQualifiedName!~TestCase2'