In what order does Karma run its tests?

Question

I'm running Karma tests through jenkins. Normally when a test fails it shows the test name and the number of the test but in some instances it just shows the number.

Does Karma run its tests in a particular order? Like alphabetically?

Below is a screenshot of my console output on jenkins.

Answer 1

It is not Karma itself that determines the order in which tests are run. The component which primarily determines the order of the tests is the test runner you decide to use with Karma. If you use Mocha, then Mocha's logic will be used. If you use Jasmine, then Jasmine's logic will be used. If you use something else, then some other logic will be used.

Mocha, for instance, runs the tests in the order in which describe and it are called in the test files. If you have two test files a.js and b.js and a.js is executed first, then Mocha will run the tests in a.js first. If b.js is executed first, then Mocha will run the tests in b.js first. There is no further sorting performed by Mocha. (Mocha has a sort option, but it is only of use when using the mocha command line tool to run Mocha in Node. It does not apply to using Mocha in Karma because Karma runs Mocha in the browser, where Mocha does not support sort.)

If you use a module loader (like RequireJS or SystemJS) to load test files dynamically, this complicates things. If you do require(["a", "b"]) and the two modules are not dependent on one another, then the order in which they load is indeterminate. a may load first, or b may load first so the order in which the tests are run may be inconsistent from one run to the next. You can force an order through configuration or by nesting require calls. (For instance, assuming again modules that are not dependent on one another,require(["a"], () => require("b")) ensures that the tests in a will execute before those in b.)

A further complication is that some test runners will abort a test run if they determine that your test suite is buggy. This does not change the order of the tests but may make it look like tests are missing. Mocha, for instance, considers errors in hooks that are used to setup and tear down test data (by opposition to errors in the test themselves) to be errors in your test suite and will abort the run. If three of your tests depend on setup code that fails, Mocha will just skip the tests. You'll see one failure out of three tests, not three failures. Here's a example test file with 3 tests and a failure in the before hook. (The before hook, aka "before all" is meant to be used to setup test data before a group of tests.)

before(() => {
    throw new Error("oh no!");
});

it("one", () => {});

it("two", () => {});

it("three", () => {});

Here's my karma.conf.js:

module.exports = function(config) {
  config.set({
    basePath: '',
    frameworks: ['mocha'],
    files: [
      'test.js'
    ],
    exclude: [
    ],
    preprocessors: {
    },
    reporters: ['progress'],
    port: 9876,
    colors: true,
    logLevel: config.LOG_INFO,
    autoWatch: false,
    browsers: ['Chrome'],
    singleRun: false,
    concurrency: Infinity
  })
}

Here's the output (I've replaced the time stamp with <ts>):

<ts>:INFO [karma]: Karma v1.7.0 server started at http://0.0.0.0:9876/
<ts>:INFO [launcher]: Launching browser Chrome with unlimited concurrency
<ts>:INFO [launcher]: Starting browser Chrome
<ts>:INFO [Chrome 58.0.3029 (Linux 0.0.0)]: Connected on socket qfNuMyp4q3SyUBOsAAAA with id 24880742
Chrome 58.0.3029 (Linux 0.0.0)  "before all" hook FAILED
        Error: oh no!
            at Context.before (test.js:3:11)
Chrome 58.0.3029 (Linux 0.0.0): Executed 1 of 3 (1 FAILED) ERROR (0.013 secs / 0.001 secs)

It says "Executed 1 of 3" because it tried executing the first test but this attempt failed in the before hook so Mocha did not try to run any of the later tests.