How to use ApprovalTests on Teamcity?

Question

I am using Approval Tests. On my dev machine I am happy with DiffReporter that starts TortoiseDiff when my test results differ from approved:

    [UseReporter(typeof (DiffReporter))]
    public class MyApprovalTests
    { ... }

However when the same tests are running on Teamcity and results are different tests fail with the following error:

System.Exception : Unable to launch: tortoisemerge.exe with arguments ...
Error Message: The system cannot find the file specified
---- System.ComponentModel.Win32Exception : The system cannot find the file 
                                                                 specified

Obviously it cannot find tortoisemerge.exe and that is fine because it is not installed on build agent. But what if it gets installed? Then for each fail another instance of tortoisemerge.exe will start and nobody will close it. Eventually tons of tortoisemerge.exe instances will kill our servers :)

So the question is -- how tests should be decorated to run Tortoise Diff on local machine and just report errors on build server? I am aware of #IF DEBUG [UseReporter(typeof (DiffReporter))] but would prefer another solution if possible.

Answer 1

There are a couple of solutions to the question of Reporters and CI. I will list them all, then point to a better solution, which is not quite enabled yet.

1. Use the AppConfigReporter. This allows you to set the reporter in your AppConfig, and you can use the QuietReporter for CI. There is a video here, along with many other reporters. The AppConfigReporter appears at 6:00. This has the advantage of separate configs, and you can decorate at the assembly level, but has the disadvantage of if you override at the class/method level, you still have the issue.

2. Create your own (2) reporters. It is worth noting that if you use a reporter, it will get called, regardless as to if it is working in the environment. IEnvironmentAwareReporter allows for composite reporters, but will not prevent a direct call to the reporter. Most likely you will need 2 reporters, one which does nothing (like a quiet reporter) but only works on your CI server, or when called by TeamCity. Will call it the TeamCity Reporter. And One, which is a multiReporter which Calls teamCity if it is working, otherwise defers to .

3. Use a FrontLoadedReporter (not quite ready). This is how ApprovalTests currently uses NCrunch. It does the above method in front of whatever is loaded in your UseReporter attribute. I have been meaning to add an assembly level attribute for configuring this, but haven't yet (sorry) I will try to add this very soon.

Hope this helps. Llewellyn

Answer 2

I recently came into this problem myself.

Borrowing from xunit and how they deal with TeamCity logging I came up with a TeamCity Reporter based on the NCrunch Reporter.

public class TeamCityReporter : IEnvironmentAwareReporter, IApprovalFailureReporter
{
    public static readonly TeamCityReporter INSTANCE = new TeamCityReporter();

    public void Report(string approved, string received) { }

    public bool IsWorkingInThisEnvironment(string forFile)
    {
        return Environment.GetEnvironmentVariable("TEAMCITY_PROJECT_NAME") != null;
    }
}

And so I could combine it with the NCrunch reporter:

public class TeamCityOrNCrunchReporter : FirstWorkingReporter
{
    public static readonly TeamCityOrNCrunchReporter INSTANCE = 
        new TeamCityOrNCrunchReporter();

    public TeamCityOrNCrunchReporter()
        : base(NCrunchReporter.INSTANCE,
        TeamCityReporter.INSTANCE) { }
}

[assembly: FrontLoadedReporter(typeof(TeamCityOrNCrunchReporter))]

Answer 3

I just came up with one small idea.

You can implement your own reporter, let's call it DebugReporter

public class DebugReporter<T> : IEnvironmentAwareReporter where T : IApprovalFailureReporter, new()
{
    private readonly T _reporter;

    public static readonly DebugReporter<T> INSTANCE = new DebugReporter<T>();

    public DebugReporter()
    {
        _reporter = new T();
    }

    public void Report(string approved, string received)
    {
        if (IsWorkingInThisEnvironment())
        {
            _reporter.Report(approved, received);
        }
    }

    public bool IsWorkingInThisEnvironment()
    {
#if DEBUG
        return true;
#else
        return false;
#endif
    }
}

Example of usage,

[UseReporter(typeof(DebugReporter<FileLauncherReporter>))]
public class SomeTests
{
    [Test]
    public void test()
    {
        Approvals.Verify("Hello");
    }
}

If test is faling, it still would be red - but reporter would not came up.

The IEnvironmentAwareReporter is specially defined for that, but unfortunatelly whatever I return there, it still calls Report() method. So, I put the IsWorkingInThisEnvironment() call inside, which is a little hackish, but works :)

Hope that Llywelyn can explain why it acts like that. (bug?)