How to run Dojo DOH unit-tests through Jenkins?

Question

Has anyone tried integrating Dojo DOH unit-tests with Jenkins?

I'd like to do the following, but don't want to reinvent it if this has already been done. So, I'm thinking:

1. Kick off the DOH-tests from a post-build step in Jenkins and wait for the results
2. Run the tests themselves in a headless-browser (e.g. Crowbar)
3. Parse the succes/error-count from the HTML returned by Crowbar
4. Find (or write) a Jenkins plugin that will (a) fail the build if there are failing tests, (b) render the test results, (c) possibly integrate results into the CI game plugin

Questions:

1. Has this been done before?
2. Do you see any issues with the outline above?
3. Do you know of a Jenkins plugin that will help, or will I have to build my own?

Answer 1

1. Automated Dojo testing - DOH & Selenium-RC (Rob Coup - 2008/01/03)

Plan:

• Have a config file defining which browsers to launch, which machines they're on, and what tests to run.
• Launch each browser via Selenium-RC
• Run the tests via the normal DOH browser runner.
• Use Selenium to extract the results from DOH.
• Collate the results from the various browsers and produce something useful.

Solution:

• Drop seleniumRunner.js, seleniumRunner.config.js, seleniumRunner.sh (or the .bat if you're on Windows), and selenium-java-client-driver.jar into util/doh/ in your Dojo install.
• Put selenium-server.jar on each test machine, then run java -jar selenium-server.jar -multiWindow so it listens for the browser-control messages.
• Edit seleniumRunner.config.js and change browsers and rootUrl to match your setup. The rootUrl needs to be reachable from each test machine.
• run ./seleniumRunner.sh seleniumRunner.config.js from util/doh/ on your workstation
• It'll load the config, fire up the browsers on each machine, run the unit tests from Dojo core, and print the pass/fail/error stats for each.
• Each browser is kicked off and monitored in a separate thread (not strictly necessary but too cool to resist doing).

Issues:

• unless I ran the selenium server in multiWindow mode Safari and Firefox would pop up Print dialogs (!?!) whenever the test page was loaded. But Safari never initialised the test page if it was in multiWindow mode. On OSX and Windows. gah.
• Opera on OSX didn't set up the Selenium proxy properly (localhost:4444 for reference).
• IE didn't like doing a dojo.connect() via the selenium javascript commands for some reason.

2. Seems reasonable to me.

3. Jenkins Selenium plugin

This plugin turns your Jenkins cluster into a Selenium2 Grid cluster, so that you can utilize your heterogeneous Jenkins clusters to carry out Selenium tests. This plugin is a turn-key solution — no additional installation nor configuration is necessary to make it work. The plugin installs Selenium Grid on all the slaves automatically and set up a grid on its own.

Answer 2

For running the D.O.H tests i have developed a tool which integrates into ci and can start the browser.

http://codeblog.bigbrowser.net/dojo-testing-d-o-h-with-continuous-integration/

Maybee you can give this also a try.

I have explained there where to download and how to run it.

Answer 3

Here's how I did it with HTMLUnit. No Selenium required.

It runs as a regular JUnit test (which can easily be run automatically by your CI Server), and prints out the DOH log if there is a test failure.

public class JavascriptTest {

  private static final int MAX_RUNNING_TIME = 10 * 1000;

  //The test runner
  public static final String PATHNAME = "src/main/webapp/library/mystuff/dojo/util/tests/runTests.html";

  //Runs all of the Dojo Objective Harness (D.O.H.) javascript tests.
  //The tests are currently grouped into test modules, and the parent module is "util.tests.module" (in module.js)
  //As you can see in the URL pathname, we pass that module name to the testRunner and it runs all the javascript tests.
  @Test
  public void runAllJavascriptTests() throws Exception {
    final WebClient webClient = new WebClient(BrowserVersion.INTERNET_EXPLORER_8);
    final HtmlPage page = webClient.getPage("file://" + new File(PATHNAME).getAbsolutePath());

    waitForTestsToRun(webClient, page);

    String log = page.getElementById("logBody").asText();
    assertTrue(log, page.asText().contains("WOOHOO!!")); //D.O.H. will display WOOHOO!! if all tests are successful.
  }

  private void waitForTestsToRun(WebClient webClient, HtmlPage page) {
    webClient.waitForBackgroundJavaScript(500);
    int runningTime = 0;
    while(testsAreRunning(page) && runningTime < MAX_RUNNING_TIME){
      webClient.waitForBackgroundJavaScript(500);
      runningTime += 500;
    }
  }

  private boolean testsAreRunning(HtmlPage page) {
    //Check if the "Tests Running" div is visible.
    return "".equals(page.getElementById("playingMsg").getAttribute("style"));
  }

}

And below is the content of runTests.html. It basically just redirects to the DOJO test runner, with parameters specific to the tests in the directory we want to test.

It's just a nice way to structure things, you could alternatively have specified this URL in the PATHNAME field in the JUnit test.

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.0 Transitional//EN">
<html>
    <head>
    <title>Dojox Unit Test Runner</title>
      <!--The "testModule" param tells the runner which test module to run-->
      <!--The "paths" param adds our dojo module paths, otherwise it would just look in the default dojo modules for code to test.-->
    <meta http-equiv="REFRESH" content="0;url=../../../../dojo-release-1.7.2-src/util/doh/runner.html?testModule=util.tests.module&paths=util,../../mystuff/dojo/util;mystuff,../../mystuff/dojo"></HEAD>
    <BODY>
        Redirecting to D.O.H runner.
    </BODY>
</HTML>