MSBuild targets to run all tests, even if some fail

Question

I have an MSBuild script that runs NUnit unit tests, using the console runner. There are multiple tests projects and I'd like to keep them as separate MSBuild targets, if possible. If the tests fail I want to overall build to fail. However, I want to continue running all the tests, even if some of them fail.

If I set ContinueOnError="true" then the build succeeds regardless of test outcomes. If I leave it at false then the build stops after the first test project that fails.

Answer 1

One way to do this would be to set the ContinueOnError="true" for the NUnit tasks but grab the exit code of the from the NUnit process. If the exit code is ever != to 0 create a new property that you can use later on in the script to fail the build.

Example:

<Project DefaultTargets="Test"
         xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <ItemGroup>
    <UnitTests Include="test1">
      <Error>true</Error>
    </UnitTests>
    <UnitTests Include="test2">
      <Error>false</Error>
    </UnitTests>
    <UnitTests Include="test3">
      <Error>true</Error>
    </UnitTests>
    <UnitTests Include="test4">
      <Error>false</Error>
    </UnitTests>
    <UnitTests Include="test5">
      <Error>false</Error>
    </UnitTests>
  </ItemGroup>

  <Target Name="Test" DependsOnTargets="RunTests">
    <!--Fail the build.  This runs after the RunTests target has completed-->
    <!--If condition passes it will out put the test assemblies that failed-->
    <Error Condition="$(FailBuild) == 'True'"
           Text="Tests that failed: @(FailedTests) "/>
  </Target>

  <Target Name="RunTests" Inputs="@(UnitTests)" Outputs="%(UnitTests.identity)">
    <!--Call NUnit here-->
    <Exec Command="if %(UnitTests.Error) == true exit 1" ContinueOnError="true">
      <!--Grab the exit code of the NUnit process-->
      <Output TaskParameter="exitcode" PropertyName="ExitCode" />
    </Exec>

    <!--Just a test message-->
    <Message Text="%(UnitTests.identity)'s exit code: $(ExitCode)"/>

    <PropertyGroup>
      <!--Create the FailedBuild property if ExitCode != 0 and set it to True-->
      <!--This will be used later on to fail the build-->
      <FailBuild Condition="$(ExitCode) != 0">True</FailBuild>
    </PropertyGroup>

    <ItemGroup>
      <!--Keep a running list of the test assemblies that have failed-->
      <FailedTests Condition="$(ExitCode) != 0"
                   Include="%(UnitTests.identity)" />
    </ItemGroup>
  </Target>

</Project>