Does the MSBuild Exec task search STDOUT for the string "error"?

Question

I have a small NUnit test suite with a single ignored test. I'm just now writing a simple MSBuild script to restore and publish and the like. I tried to get a dotnet test task working

<Target Name="test" DependsOnTargets="restore">
    <Exec Command="dotnet test"
          WorkingDirectory="test\Example.Tests" />
</Target>

But, it exits with code -1 every time!

PS> dotnet msbuild /t:test build.xml
Microsoft (R) Build Engine version 15.3.409.57025 for .NET Core
Copyright (C) Microsoft Corporation. All rights reserved.

    Restore completed in 50.88 ms for C:\...\src\Example\Example.csproj.
    Restore completed in 57.18 ms for C:\...\test\Example.Tests\Example.Tests.csproj.
  Build started, please wait...
  Build completed.

  Test run for C:\...\test\Example.Tests\bin\Debug\netcoreapp1.1\Example.Tests.dll(.NETCoreApp,Version=v1.1)
  Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
  Copyright (c) Microsoft Corporation.  All rights reserved.

        Starting test execution, please wait...
  NUnit Adapter 3.8.0.0: Test execution started
  Running all tests in C:\...\test\Example.Tests\bin\Debug\netcoreapp1.1\Example.Tests.dll
  NUnit3TestExecutor converted 26 of 26 NUnit test cases
  Skipped  FilterEndpointWithCompletelyFilteredSystemsReturnsNotFound("foo")
EXEC : error Message:  [C:\...\build.xml]
   OneTimeSetUp: Having trouble constructing this scenario with the current command catalog

  NUnit Adapter 3.8.0.0: Test execution complete

  Total tests: 26. Passed: 25. Failed: 0. Skipped: 1.
  Test Run Successful.
  Test execution time: 2.8322 Seconds


C:\...\build.xml(16,9): error MSB3073: The command "dotnet test" exited with code -1.

If I run the command directly in my console, it's fine.

PS> dotnet test
Build started, please wait...
Build completed.

Test run for C:\...\test\Example.Tests\bin\Debug\netcoreapp1.1\Example.Tests.dll(.NETCoreApp,Version=v1.1)
Microsoft (R) Test Execution Command Line Tool Version 15.0.0.0
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
NUnit Adapter 3.8.0.0: Test execution started
Running all tests in C:\...\test\Example.Tests\bin\Debug\netcoreapp1.1\Example.Tests.dll
NUnit3TestExecutor converted 26 of 26 NUnit test cases
Skipped  FilterEndpointWithCompletelyFilteredSystemsReturnsNotFound("foo")
Error Message:
 OneTimeSetUp: Having trouble constructing this scenario with the current command catalog

NUnit Adapter 3.8.0.0: Test execution complete

Total tests: 26. Passed: 25. Failed: 0. Skipped: 1.
Test Run Successful.
Test execution time: 3.6485 Seconds

I searched around for help, but didn't quite find anything explicit. I got the impression that the Exec task searches STDOUT for some kind of error message, possibly just the word "error", in order to set the exit code/status.

This does appear on my console (for some reason NUnit prints "Error Message" for ignored/skipped tests).

Skipped  FilterEndpointWithCompletelyFilteredSystemsReturnsNotFound("foo")
Error Message:
 OneTimeSetUp: Having trouble constructing this scenario with the current command catalog

If I comment out this test, the run passes (via msbuild).

Am I correct about the Exec task? Would I "fix" the problem by overriding the CustomErrorRegularExpression parameter? I can't find any good info about this parameter... would I set it to an empty string?

Answer 1

If provided, Exec checks the output against CustomErrorRegularExpression and CustomWarningRegularExpression first. If those do not match or were not provided, Exec then uses the following RegEx:

            new Regex
            (
            // Beginning of line and any amount of whitespace.
            @"^\s*"
                // Match a [optional project number prefix 'ddd>'], single letter + colon + remaining filename, or
                // string with no colon followed by a colon.
            + @"(((?<ORIGIN>(((\d+>)?[a-zA-Z]?:[^:]*)|([^:]*))):)"
                // Origin may also be empty. In this case there's no trailing colon.
            + "|())"
                // Match the empty string or a string without a colon that ends with a space
            + "(?<SUBCATEGORY>(()|([^:]*? )))"
                // Match 'error' or 'warning'.
            + @"(?<CATEGORY>(error|warning))"
                // Match anything starting with a space that's not a colon/space, followed by a colon.
                // Error code is optional in which case "error"/"warning" can be followed immediately by a colon.
            + @"( \s*(?<CODE>[^: ]*))?\s*:"
                // Whatever's left on this line, including colons.
            + "(?<TEXT>.*)$",
            RegexOptions.IgnoreCase | RegexOptions.Compiled
            )

That works out lines in the format of the following example being interpreted as errors or warnings (depending which of those two words are in the "Cat." part).

Main.cs(17,20):Command line warning CS0168: The variable 'foo' is declared but never used
-------------- ------------ ------- ------  ----------------------------------------------
Origin         SubCategory  Cat.    Code    Text

Would I "fix" the problem by overriding the CustomErrorRegularExpression parameter?

EDIT

(Question for myself: Why on Earth did I say "Yup" initially when my second sentence directly contradicts this?)

Nope. :) Set IgnoreStandardErrorWarningFormat to true. From the documentation you linked in your question:

IgnoreStandardErrorWarningFormat: Optional Boolean parameter.

• If false, selects lines in the output that match the standard error/warning format, and logs them as errors/warnings.
• If true, disable this behavior.

The default value is false.