How to automate non-interactive tests on Microsoft Surface

Question

We have a suite of tests (test cases for a native library) that we want to run on Microsoft Surface (ARM version). It is easy to do it by hand from Visual Studio. However, we would like to automate the process in a way that we can compile, install, run and collect the output from command line (e.g. from Jenkins CI).

On iOS there is Fruitstrap that emulates XCode to do similar task for iOS. Is there a Microsoft/third-party tool that could help us do this for Surface?

The question about automating GUI testing (Best way to test a Microsoft Surface application) is not relevant in this case because we don't have any GUI and the problem is to upload and start the application on the Surface from the host non-interactively.

Edit: There is a nice video of how one would use Google Test framework in VS2012 http://www.youtube.com/watch?v=o-Gi6REeGN0 but our problem is that we want to run the tests on Microsoft Surface instead of the host.

Answer 1

I've used psexec to run tests on specific environments before and it appears that psexec runs on Surface:

http://technet.microsoft.com/en-us/library/ee692107(v=surface.10).aspx

I'm not sure that you can run mstest.exe or nunit.exe on the surface, so if not I would have made my own light weight test framework that would produce some kind of easily parsed output (probably xml) with test results and put it together in an .exe file.

So your jenkins build script would have steps like this:

[Copy SurfaceTests.exe to \\mySurfaceDevice\c$\testfolder]
psexec.exe \\mySurfaceDevice c:\testfolder\surfacetests.exe  > c:\testfolder\output.xml
[Copy \\mySurfaceDevice\c$\testfolder\output.xml to jenkins folder]

then you would have to configure jenkins to parse your output. If you investigate the output of nunit or mstest results you can probably produce something similar in your surfacetests.exe and set up jenkins as if it was produced by one of those. I don't have any experience with Jenkins, but in cruisecontrol.net it is quite easy to make xslt files that parsed xml output and presented in build reports.

If psexec approach does not work, I would have made a small wcf service host that was running allways on the device. This service has one method (RunTests()) that executed your surfacetests.exe whenever it was invoked. Or better still, maybe just run the tests from an assembly stored on disk (you would have to do some reflection do achieve this), and returned the results in the method call. Here's an article that explains how to create a service that will run on windows store apps

http://www.c-sharpcorner.com/uploadfile/7e39ca/simple-wcf-service-in-windows-store-apps/

Good luck. Sounds like an interesting task. Please update on how you ended up solving this.