Mocha + Chai command line exit with error code number equal test fail count

Question

I am really new to Node, and am trying to create a stand alone program. I am using Mocha and Chai to test my functions (new at this as well), and I run them with this node run script:

"test": "mocha tools/testSetup.js \"src/**/*.spec.js\" --reporter progress"

(I tried without the reporter option as well). Tests run fine, when a test fails it keeps running the rest, and then creates a (quite verbose) report of what failed. But then the test suit exits node with an error code equals to the number of tests that failed, so if three tests failed then the error code is 3, if no tests failed then the error code is 0. This makes an awful output when at least one test fails, similar to when the application crashes.

Is this by design? am I supposed to do something to "catch" the error? Or am I doing something wrong and the exit code should actually be just 0 regardless of tests failing?

Answer 1

Is this by design?

Yes, it is, both on npm's part and Mocha's part.

Normally, when you run a test suite, you want the test runner to have a non-zero exit code when there's any failure. Why? The non-zero exit code makes it trivially easy for the code that invokes the test runner to know whether the tests passed or not. There's no need to parse a report, or anything else. Just check the exit code. For cases where the code that invokes the test runner does not care, then it can just ignore the exit code.

Mocha happens to exit with the number of failures, which satisfies the minimal requirement that a non-zero exit code be used when there is any failure.

From the perspective of npm, the hard failure you get, which looks like a crash, is also on purpose. The issue is that npm is often used for installing packages, and some packages perform a test run as part of their installation. You want a failure there to be a hard failure. Perhaps npm could be designed to distinguish the user issuing a run-of-the-mill npm test at the command line, from a test run as part of an install. This would be an enhancement to npm but the current behavior is not happenstance.

am I supposed to do something to "catch" the error?

If what you want to do is prevent npm from giving you an ELIFECYCLE error, you could neutralize the error by doing mocha [parameters] || true but this would effectively make anything that invokes npm test unable to detect immediately, through the exit code, whether the tests passed or not. Any tool you may use that depends on npm test would be unable to know whether the tests passed or not. I would not do this.

Or am I doing something wrong and the exit code should actually be just 0 regardless of tests failing?

No, as explained above, the exit code from Mocha is by design. If you were to modify your test suite to try to work around it, you'd be breaking a feature.

Personally, I sidestep the whole issue of npm screaming the sky is falling when a test fails by invoking my test suite outside npm (e.g. most often this means running gulp test because I use gulp to run build tasks), and I reserve npm test for those situations where a test failure is really a critical failure.