TravisCI is not failing my build when tests fail

Question

I have tests written in JavaScript and I use TravisCI for tests.

Setup

My package.json is something like:

"scripts": {
  "test": "node testsRunner.js"
}

And my .travis.yml is:

language: node_js
node_js:
- '0.12.7'

'testsRunner.js' is:

var nodeunit = require('nodeunit');
var path = require('path');

nodeunit.reporters.default.run([
  path.join(__dirname, 'suite1/test.js')
]);

And suite1/test.js finally is:

module.exports = {
  setUp: function(callback) {
    // Initialization code...
    callback();
  },

  tearDown: function(callback) {
    // Cleanup...
    callback();
  }, 

  test1: function(test) {
    test.expect(10); // This test expects 10 assertions to be run
    // Doing stuff...
    test.done();
  },

  test2: function(test) {
    test.expect(10); // This test expects 20 assertions to be run
    // Doing stuff...
    test.done();
  }
};

Log from Travis

Here is the build log and test execution from Travis:

Using worker: worker-linux-docker-19fc8ef0.prod.travis-ci.org:travis-linux-6
system_info
Build system information
Build language: node_js
Build image provisioning date and time
Thu Feb  5 15:09:33 UTC 2015
Operating System Details
Distributor ID: Ubuntu
Description:    Ubuntu 12.04.5 LTS
Release:    12.04
Codename:   precise
Linux Version
3.13.0-29-generic
Cookbooks Version
a68419e https://github.com/travis-ci/travis-cookbooks/tree/a68419e
GCC version
gcc (Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3
Copyright (C) 2011 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
LLVM version
clang version 3.4 (tags/RELEASE_34/final)
Target: x86_64-unknown-linux-gnu
Thread model: posix
Pre-installed Ruby versions
ruby-1.9.3-p551
Pre-installed Node.js versions
v0.10.36
Pre-installed Go versions
1.4.1
Redis version
redis-server 2.8.19
riak version
2.0.2
MongoDB version
MongoDB 2.4.12
CouchDB version
couchdb 1.6.1
Neo4j version
1.9.4
RabbitMQ Version
3.4.3
ElasticSearch version
1.4.0
Installed Sphinx versions
2.0.10
2.1.9
2.2.6
Default Sphinx version
2.2.6
Installed Firefox version
firefox 31.0esr
PhantomJS version
1.9.8
ant -version
Apache Ant(TM) version 1.8.2 compiled on December 3 2011
mvn -version
Apache Maven 3.2.5 (12a6b3acb947671f09b81f49094c53f426d8cea1; 2014-12-14T17:29:23+00:00)
Maven home: /usr/local/maven
Java version: 1.7.0_76, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-7-oracle/jre
Default locale: en_US, platform encoding: ANSI_X3.4-1968
OS name: "linux", version: "3.13.0-29-generic", arch: "amd64", family: "unix"
git.checkout
0.61s$ git clone --depth=50 --branch=master git://github.com/myuser/myproj.git myuser/myproj
Cloning into 'myuser/myproj'...
remote: Counting objects: 1473, done.
remote: Compressing objects: 100% (1053/1053), done.
remote: Total 1473 (delta 300), reused 1401 (delta 250), pack-reused 0
Receiving objects: 100% (1473/1473), 8.93 MiB | 0 bytes/s, done.
Resolving deltas: 100% (300/300), done.
Checking connectivity... done.
$ cd myuser/myproj
$ git checkout -qf d603836a30ea1bc213f7b97682df507c91af8404
This job is running on container-based infrastructure, which does not allow use of 'sudo', setuid and setguid executables.
If you require sudo, add 'sudo: required' to your .travis.yml
See http://docs.travis-ci.com/user/workers/container-based-infrastructure/ for details.
2.99s$ nvm install 0.12.7
######################################################################## 100.0%
Now using node v0.12.7
$ node --version
v0.12.7
$ npm --version
2.11.3
$ nvm --version
0.23.3
install
0.84s$ npm install 
npm WARN package.json [email protected] No license field.
0.64s$ npm test
> [email protected] test /home/travis/build/myuser/myproj
> node testsRunner.js
test.js
✔ test1
✖ test2
TypeError: Cannot read property...
    at extract (...)
    at Object.at (...)
    at Object.module.exports...
    ... <stacktrace continues>

FAILURES: 1/45 assertions failed (19ms)
The command "npm test" exited with 0.
Done. Your build exited with 0.

The problem

Problem is that test1 succeeds and test2 does not pass. Before test2 threw exceptions and this made Travis report my build as a failure.

However now i fixed the test, so test2 does not throw exceptions, but its asserts fail. However Travis is reporting that build as a pass, while it should report it as a failed build.

How to handle this?

Edit

I have changes my runner like this:

var nodeunit = require('nodeunit');
var path = require('path');

nodeunit.reporters.default.run([
  path.join(__dirname, 'suite1/test.js')
]);
console.log('Done!'); // ADDED THIS LINE

Well, in the log I can see that Done! is displayed before everithing, after that I can see each single test log and stacktraces. Is it possible that these tests are executing async and this is causing the main process to exit with 0 status?

Fixed

Following Chris Beck's suggestion I changed the runner like this:

var nodeunit = require('nodeunit');
var path = require('path');

nodeunit.reporters.default.run([
  path.join(__dirname, 'suite1/test.js')
], null function(data) {
  // `data` has a value if errors occurred
  if (data) { throw 'Error1'; } 
});

Answer 1

I have used travis-ci for unit tests.

If you want the test failures to represent a build failure, the command-line utility that invokes the tests needs to report a nonzero error code result (as would be visible in the bash value $? after running the tests.)

If your js environment does not cause assert failures to do this, the most simple thing most likely is to write your own assert function which throws an uncaught exception, or, which sets a global variable to indicate failure, and before exist your test runner script checks the global variable and throws an uncaught exception if it is set.

(The latter has the advantage that you run all the tests even if some asserts fail, but the disadvantage that if the tests loop infinitely then your build will stall for a while even after an earlier assertion failure.)