How can I make MSTest.exe return a 0 even if a test fails?

Question

We have a build that runs on TeamCity and contains several different test projects. They each run one after the other, and we're using MSTest.exe from a Rakefile in order to orchestrate everything, thusly:

desc 'Run Unit Tests'
mstest :data_test => [:build_database_tests] do |mstest|
  puts build_header("Data Tests")
  mstest.command = msTestCommand
  mstest.parameters = [
    "/resultsfile:dTest.trx", 
    "/detail:errormessage", 
    "/detail:description",
    "/usestderr"
  ]
  mstest.assemblies  "../../../Database/DataTests/bin/Release/DataTests.dll"
end

This produces a nice TRX file that TeamCity can parse and everything, making for a nice clean report tab. However, the MSTest.exe returns a 1 if a test fails, which rake interprets as a failure, and causes the rest of the tests to not be run.

We would like to suppress this behavior; we want the build to continue to run when the test fails so that we can see what other tests may also fail. How can we get Rake to ignore the return code, and/or cause MSTest.exe to not emit a failing return code if a test fails?

Answer 1

MSTest by itself will execute ALL test sets, even if previous one has failed. Your issue is with the interpretation Rake is making. Most, if not all, build languages (ANT, MSBuild, Gradle, etc...) have a "Continue on error" like behavior.

Doing a quick Google search for "rake continue on error" I found a couple links:

http://www.rake.build/fascicles/004-ignore-failed-tasks.html Continue Rake after failure

The basic gist I got was that you can use sh to iterate over a series of tasks and ignore failures so that ALL tasks execute.