State of unit testing for Windows Phone

Question

I've been pushing my Google Fu to the limits trying to find the most recommended / stable setup for doing TDD + CI for Windows Phone applications. Can anyone who has successfully been doing this point me in the right direction?

Here's what I want to be able to do (if it's possible):

• Write unit tests for view models and application services that don't require phone functionality
• Execute tests directly in Visual Studio via Resharper or TD.NET
• Execute the unit tests from the command line with XML out, without launching the emulator
• Preferably be resiliant (as far as third party libraries go) to SDK updates

Since I'd like to keep this question as a resource to others looking for the same thing, here's what I'd prefer answers to avoid:

• Open source ports of projects that are either incomplete or abandoned
• Projects that are only available as an attachment on someone's blog

I'd also like to get full BDD-style acceptance tests going, but that's another issue entirely.

Answer 1

I'm adding this answer as Community Wiki so that others may modify it to keep it up to date.

Unit Testing

Intent: To run isolated (no phone functionality), fast-executing tests often from both the IDE and Continuous Integration server without requiring the emulator (eg. TDD View Models)

The method I've seen recommended in a number of presentations involves referencing your source files in a .NET 4 project and running the tests against those (referencing the desktop equivalents of the assemblies). If your code doesn't use any APIs that are different to the desktop BCL and you can deal with keeping the reference project up to date (new files aren't added automatically) than that should be sufficient.

Otherwise, you can follow the steps below to execute code that references WP7 assemblies in the desktop CLR:

1. Add a tests Silverlight 4 Class Library (not a Silverlight for Windows Phone project) to your solution
2. In the class library, set Copy Local for all the framework references except mscorelib (basically System.* and Microsoft.*) to true
3. Add a reference from this test libary to NUnit.Silverlight.Framework.dll and NUnit.Silverlight.Compatibility.dll from the NUnit-Silverlight project
4. Add a reference from the test libary to your main Windows Phone project, ignoring the version warning.
5. Run the tests from in the IDE using ReSharper (tested) or TestDriven.NET
6. Run the tests from the command line using the standard .NET 2.0 nunit-console.exe from the latest NUnit release, passing in /framework=v4.0.

(Many of the above workarounds are required because WP7 uses SL3. Once Mango is released with the SL4 runtime, it should be a cleaner setup)

Integration Testing

Intent: To run longer-running tests that interact with resources external to the code (like phone features and web services) on the emulator, both on demand and on the CI server

UPDATE Unit Testing Windows Phone 8 applications will be official supported in Visual Studio 2012 Update 2, including VS integration and command line support. These tests run in the emulator, so I've included it under Integration Tests.

This is not currently supported by the WP7 port of the Silverlight test framework (and that only ships as a download from a blog).

In the meantime, I have created a codeplex project that adds an MSBuild task that launches the emulator and collates the results into an XML file. The simplest method installation is to add the wp7-ci NuGet package.

NOTE: Installing the WP7 SDK on Windows Server requires modifying the installer configuration and is not supported, but works well.

Acceptance/System Testing

Intent: To run end-to-end automated tests that interact with the phone's UI on the emulator, both on demand or on the CI server

Expensify's (poorly named for SEO) Windows Phone Test Framework supports writing UI automation tests from a host PC using SpecFlow.

Answer 2

I think a matter of this has to due with how you write your tests.

• Eventually some code will have to touch some phone specific things. Those should be isolated as dependencies and faked out. I haven't figured out a proper way to get the unit tests to actually run within the phone itself. Unfortunately, those dependencies remain untested for me.
• Using NUnit for Silverlight allows you to get assertions: http://code.google.com/p/nunit-silverlight/
• I've used Resharper to run the unit tests without issue. Similarly, you can use the nunit-console to run the tests and get XML output.
• Ayende's Rhino Mocks for Silverlight work fine for mocking / stubbing dependencies.
• Continuous integration was a bit tricky. The WP7 SDK is not available for on the Server platform, so I built a new one on Windows 7 for my CI. There may be ways around that limitation, but I didn't bother.

The other tool you will want is MVVMLight. This will allow you to use EventTrigger and ICommand instead of events, since testing the events is significantly more work and can't be bound through the DataContext.

As far as how I designed my application:

The ViewModel can take in any number of dependencies, which get resolved using MicroIoC.

The actual code behind of the XAML resolved the ViewModel and sets it to the data context. This is unfortunate because that means you can't set the DataContext in XAML, but was a trade off I was willing to accept for dependency injection, like this:

public partial class SignUpPage
{
    public SignUpPage()
    {
        InitializeComponent();
        DataContext = IoC.Resolve<SignUpViewModel>();
    }
}

Fortunately, that's the only C# code that actually appears in my XAML code behind. From there, it's fairly regular MVVM with using binding and the DataContext.

Now you can test your ViewModel, inject the required dependencies (or fake them out) and it'll run fine without being in the emulator, so long as you don't try to use something WP7 specific.

Answer 3

Here is a link to someone that has managed to automate WP7 tests on the 'phone, to automatically deploy the app, run tests, and read results: http://justinangel.net/WindowsPhone7EmulatorAutomation

It makes use of the CoreCon API, which looked super interesting until I started playing with it, and discovered that most of the functions threw not-implemented exceptions - but there is enough there do run automated tests.