Why is my Node.js getting "program terminated" without an error

Question

I have a fairly large Node.js project that's been working great for an extensive period of development. Recently however -- and unfortunately cannot pinpoint exactly when it changed -- my application has started exiting with program terminated only when running from within the node.js debugger.

The big question: How do I find out what is causing this error?

Breakdown of what I know:

• Only seems to occur from within the debugger (e.g. node debug main.js)
• Same results with Node 4.2.5 & 4.2.6. With Node 5.5.0, the debugger generally just stops working when this happens.
• There is no additional output such as stack trace or error messages, only "program terminated"
• process events of SIGINT, exit, nor uncaughtException are hit
• Through console.log() I've been able to determine that this happens when I require a module (more information on this below)
• The require logic is surrounded by try/catch but it does not catch
• I have tried using a domain's error handler to catch this by putting the require portion in the run method but nothing caught there, either
• I've used tools such as madge to ensure I do not have any circular dependencies (Perhaps there is a better way to do this)

About require:

My program has a "plugin" type system in which modules are require'd at runtime as needed. The offending require is not deterministic from what I can tell, however: A module that can be require'd a few times in a row with success fails other times.

Again, this is code that has been working for months. I've been making some fairly extensive changes (but not to the area of code using require mentioned above) when I started noticing this. Since I can often run for a few minutes without issue though many different code paths before seeing this, it's hard to say what is triggering this.

What else can I do here?!

For reference, the (not quite up to date as I've not pushed my changes) code is on Github, if that helps.

Answer 1

After a lot more digging and asking around I have some answers, so I'll answer this myself:

• As for the particular error, it seems to be a Node.js bug
• As for the question of "Why am I getting this error?" read on...

The Cause

When Node.js itself encounters a segfault, the program (e.g. .js) will simply stop. You will not get an exception, error handlers executed, etc. as the underlying node has died. If you're running under the debugger, the only message you'll see is program terminated.

Finding Out Why

To find out why node is terminating a couple options work well:

1. Install node-segfault-handler.
2. Use gdb

node-segfault-handler

node-segfault-handler is a drop in module that will give you a basic call stack when something explodes. Simply require and init it at the top of your index.js or equivalent:

var SegfaultHandler = require('segfault-handler');
SegfaultHandler.registerHandler("crash.log");

See the authors page for additional information and usage.

gdb

With gdb you can get more readable stack information and truly debug if needed. Note that a common cause for Node.js segmentation faults is native module(s) you may be using. If this is the case, it's probably a good idea to (re)build them with debug symbols so your stack trace has something useful. Native modules are generally using node-gyp. In this case, find the modules binding.gyp file's cflags member and add -g to it. Example from the sqlite3 package:

 "cflags": [ "-include ../src/gcc-preinclude.h", "-g" ],

Next rebuild the module(s) you updated:

npm rebuild --build-from-source sqlite3

Finally, execute node and your program under gdb or attach it. To debug the debugger (like in my case) the steps would be something like this:

1. In the shell, run your node app with gdb --args in front, e.g. gdb --args node debug main.js
2. At the gdb prompt, enter run
3. Wait for the crash. Once you hit it, type bt to get the stack information, or debug away!