One failing test causes other async tests to fail

Question

I have a very basic karma/jasmine setup with one test suite containing two tests. I expect the first test to fail and the second test to pass.

describe("The system", function() {

    it("should fail", function() {
        expect(true).toBe(false);
    });

    it("should succeed", function(done) {
        setTimeout(function() {
            expect(1).toBe(1);
            done();
        }, 10);
    });
});

However, when I run these tests in the browser and click the Debug button to open the Karma DEBUG RUNNER, I see both tests failing, where the second test fails with the error message of the first test. The regular test run (i.e. not in the Karma DEBUG RUNNER) works as expected.

The error message for the second test is:

Uncaught Expected true to be false.
    at UserContext.<anonymous> (http://localhost:9876/base/workingspec.js:4:22) thrown

When I disable or remove the first test, the second test passes.

Why do both tests fail in this case? Why does the second test fail with the error message of the first test?

---

My test setup contains the following packages/versions:

+-- [email protected]
+-- [email protected]
+-- [email protected]
+-- [email protected]
`-- [email protected]

Answer 1

The problem is indeed in the Debug.js file of the Karma Debug Runner, as @user907860 already hinted at. It is not particular to Jasmine. I have reported the issue and created a fix which has just been merged into the master branch of Karma, so the next release should fix this issue.

Answer 2

At first I thought it was a bug, but after some research with the brilliant Chrome devtools, it seems that this is an expected behavior, at least by Jasmine. It may be, though, a bug in the Karma framework.

In short, the node_modules/karma/static/debug.js file (which is the js-file for the debug page) has these lines (I have Karma v1.7.0):

for (var i = 0; i < result.log.length; i++) {
  // Throwing error without losing stack trace
  (function (err) {
    setTimeout(function () {
      throw err
    })
  })(result.log[i])
}

If you comment the throw line, and restart the karma server, you'll see only console log messages, which should be expected: first FAIL, then PASS, and then the summary.

---

Verbosely, the bug in Karma may be in it's behavior to report after each spec.

That is what's happening here step by step (I have version "jasmine-core": "^2.6.4", at least this is in my package.json file):

1. Jasmine runs the first spec and it fails;
2. Karma reports on this in the log and adds a function, which throws an error to the stack (I assume the reader knows the asynchronous model in JavaScript, if not, then he must read about this in something like "Effective JavaScript: 68 Specific Ways to Harness the Power of JavaScript" by David Herman, a true gemstone book, or elsewhere). Also, this is important, though I'm not sure, since I haven't look into the code so deep, it registers sort of "globalError", since on the next step
3. Jasmine runs the second spec, and calls the getJasmineRequireObj().GlobalErrors function (jasmine.js:2204). The "globalError" is detected and the spec instantly becomes a failure. The asynchronous expect is added to the stack after Karma's function, the one, which throws the error

4. Then the first function (which throws the error) starts execution, add it throws. This function will always be called before the Jasmine asynchronous spec, since in the debug.js there is no time passed to the setTimeout call:

   //notice the absent time argument setTimeout(function () { throw err })

5. Jasmine runs the assync expect from the second spec and it passes

6. Karma reports on the second spec as a failure and adds the error throwing (if not commented out, then no error occurs and this spec passes) to the stack
7. Karma's second error is thrown

Below are screen shots with numbers, put by me to illustrate the steps from the list:

with throw not commented in the debug.js file:

and with the commented throw: