How to automate unit tests for Windows 8 HTML5 apps?

Question

I'm writing a Windows 8 app, and writing unit tests for it. How can I run them, in an automated fashion, inside the Windows 8 environment?

If I write super-simple JavaScript files, with no dependencies on the Windows 8 environment or the DOM, I can unit test them from the command line with Node.js. This is very fast (less than one second).

If I need to involve the DOM, and I keep my DOM usage to not-too-new features, I can use jsdom in Node.js and get the same setup and the same speed.

But jsdom is incomplete, and oftentimes I want to use basic Windows 8 features like the WinJS base library (e.g. WinJS.Promise) or the Windows.* enumerations. And ideally I should be testing in the same JS engine and DOM environment as my app will actually run in.

So ideally I need a test runner: a lightweight Windows 8 HTML5 container that can run some unit tests with a real DOM, and access to the WinJS and Windows Runtime APIs. For integration into the build process, I also need the ability to report results back to the command line (stdout, stderr) and change my test runner's return code depending on success or failure. And it should be completely automated, runnable in the background, and very fast (less than 10 seconds).

I know there are tools like this for WebKit, e.g. PhantomJS. Does such a thing exist for the Windows Runtime HTML5 environment? If not, what APIs should I look into for building something like it?

Answer 1

Currently there isn't a way to run a Win8/WinJS test suite in a headless manner. The runtime required for WinJS apps can't be stood up on it's own, without running a full WinJS app.

The best way to run tests, IME, is to create a separate WinJS project in your solution to run your tests.

Christopher Bennage has a blog post that describes the basic set up, here: http://dev.bennage.com/blog/2012/08/15/unit-testing-winjs/

And I have a few blog posts that head a little further down the path:

http://lostechies.com/derickbailey/category/winjs/

http://lostechies.com/derickbailey/2012/08/17/asynchronous-unit-tests-with-mocha-promises-and-winjs/

http://lostechies.com/derickbailey/2012/08/21/a-winjs-specrunner-automating-script-tag-insertion-for-unit-tests/

If you don't like the idea of running a separate project for your tests, or if you want QUnit instead of Jasmine or Mocha, check out QUnit-Metro: http://qunitmetro.github.com/QUnitMetro/ - it runs in-project with your real app, and gives you a context menu in development mode to run the tests for the page you're on.

I'm not a fan of QUnit or the way QUnit-Metro works in-page. But you might find it suitable if you don't like a separate project.

Hope that helps.