Why is UnitTestOutcome set to Unknown when running tests on TeamCity?

Question

I am checking TestContext.CurrentTestOutcome in my TestCleanup method in order to perform an action if the test did not pass (in this case, the tests are using Selenium to exercise a website and I am saving a screenshot if the test does not pass).

private static TestContext _testContext;

private static IWebDriver _driver;

[ClassInitialize]
public static void SetupTests(TestContext testContext)
{
    _testContext = testContext;
    _driver = new FirefoxDriver();
}

[TestCleanup]
public void TeardownTest()
{
    if (_testContext.CurrentTestOutcome != UnitTestOutcome.Passed)
    {
        var fileName = Path.Combine(
            Environment.CurrentDirectory,
            string.Format("{0}.{1}.gif", _testContext.FullyQualifiedTestClassName, _testContext.TestName));

        ((ITakesScreenshot)driver).GetScreenshot().SaveAsFile(fileName, ImageFormat.Gif);

        Console.WriteLine("Test outcome was {0}, saved image of page to '{1}'", _testContext.CurrentTestOutcome, fileName);
    }
}

This works well when run on a local development PC using ReSharper, but on our build server (which uses TeamCity) the UnitTestOutcome is always Unknown, although TeamCity reports them as passed.

The documentation on MSDN is not very helpful. What can cause this value to be set to Unknown?

Answer 1

According to http://confluence.jetbrains.com/display/TCD8/MSTest+Support TeamCity does not support on-the-fly reporting of individual test results, it parses the tests results file to provide the results to the build step.

That would explain how TeamCity is able to report the tests as passed even though UnitTestOutcome may be unknown at the time an individual test has completed.

The link above mentions "specifics of MSTest tool" as the reason for non-on-the-fly test result reporting so I can only theorize that the same specifics may mean that TestContext is unavailable when running from your build server.

Also, the MSDN documentation for TestContext.CurrentTestOutcome does mention that Full Trust for the immediate caller is required. TeamCity could be executing the tests in a manner that is only partially trusted and therefore causing the test outcome to be Unknown.

A quick way to check if MSTest is your problem would be to switch to NUnit using:

#if NUNIT
using NUnit.Framework;
using TestClass = NUnit.Framework.TestFixtureAttribute;
using TestMethod = NUnit.Framework.TestAttribute;
using TestInitialize = NUnit.Framework.SetUpAttribute;
using TestCleanup = NUnit.Framework.TearDownAttribute;
using IgnoreAttribute = NUnit.Framework.IgnoreAttribute;
#else
using Microsoft.VisualStudio.TestTools.UnitTesting;
using IgnoreAttribute = Microsoft.VisualStudio.TestTools.UnitTesting.IgnoreAttribute;
#endif

• source http://www.anotherchris.net/tools/using-team-city-for-staging-and-test-builds-with-asp-net-and-selenium/

You would have to do something similar in your TeardownTest method to use the NUnit TestContext.CurrentContext.Result.Status though.

Answer 2

The fix for this issue is to use a public property for TestContext, rather than using the parameter passed to the [ClassInitialize] method.

i.e.

public TestContext TestContext { get; set; }

The test runner will automatically set the property.

(This is related to another question I posted on SO)