How Node.js silently crashes without any information?

Question

Application silently dies when it opens a lot of SFTP connections. I use all sorts of process event handlers and the only one triggered is process.on('exit',...

I'm using an npm package for SFTP ssh2-sftp-client which is using ssh2 package underneath.

Here is a minimal reproducible code

const Client = require('ssh2-sftp-client')
const inspect = require('util').inspect

const sftpClient = new Client();

sftpClient.on('error', (err) => {
  console.log('SFTP client emitted ERROR: ' + inspect(err))
})
sftpClient.on('end', () => {
  console.log('SFTP client emitted END')
})
sftpClient.on('finish', () => {
  console.log('SFTP client emitted FINISH')
})
sftpClient.on('close', () => {
  console.log('SFTP client emitted CLOSE')
})

const options = {
  host: '',
  port: '22',
  username: '',
  password: ''
};

// below infinite loop is to simulate large number of 
// connections in my production code
(async () => {
  while (true) {
    try {
      await new Promise(resolve => setTimeout(() => resolve(sftpClient.connect(options)), 1000))
    } catch {
      console.log('boom')
    }
  }
})()

process.on('exit', code => console.log(`Caught!. Exit code: ${code}`))

process.on('uncaughtException', (err, origin) => {
  console.error('Unhandled exception. Please handle!', err.stack || err)
  console.error(`Origin: ${JSON.stringify(origin)}`)
})

process.on('unhandledRejection', (err, promise) => {
  console.error('Unhandled promise rejection. Please handle!', promise, err.stack || err)
})

process.on('warning', (warning) => {
  console.warn(warning.name)
  console.warn(warning.message)
  console.warn(warning.stack)
})

process.on('rejectionHandled', (promise) => {
  console.log('rejectionHandled event triggered')
})

As you can see, it shouldn't escape the infinite loop unless there is an Error. And it in fact does not for some iterations but eventually it escapes (usually after <10 iterations). And the only log I see is

SFTP client emitted END
Caught!. Exit code: 0

On the SFTP server side I have the following limitations in the sshd_config file

MaxSessions 1
MaxStartups 1

I'm using:

• Node.js 12
• ssh2-sftp-client 4.1.0

In brief, question:

How can Node crash without any error events/stacktraces and exit code 0?

Update #1

It was recommended in the comments to make a core dump of the process and analyse it.

How can I make a core dump on exit code 0?

If I make a core dump, what should I look for there?

Update #2

Created an issue in the repository https://github.com/theophilusx/ssh2-sftp-client/issues/168

Answer 1

This specific issue was being caused by the combination of the remote server dropping the TCP connection and the SSH2 library only raising an 'end' event with no 'error' event. This has been flagged as a bug in ssh2 and is listed in the issues for that module.

In the meantime, a work-around has been added to ssh2-sftp-client. Essentially, an additional 'end' listener has been added during the connection stage which will result in the connect promise being rejected if there is an end event raised. A new version of the package (v4.2.0) has been released which includes this fix.

For background information - part of the reason it is difficult to get consistent results when tracking down this issue is partly due to the MaxStartups setting of many SFTP servers (like openSSH). This setting can be a tuple of the form max:drop:full where max is the maximum allowed unauthenticated connections (connections which have not yet completed the handshake and authentication process), drop - percentage of connections to begin dropping after max is reached and full = number of unauthenticated connections to start dropping all connection attempts. The default value is often 10:30:60, which means allow 10 unauthenticated connections, start dropping 30% of them once there are more than 10 and drop all attempts once 60 unauthenticated connections occur. The drop value means that once more than 10 unauthenticated connections exist, attempts will be droped 30% of the time, which creates a somewhat random result in that each time you run your tests, different connections may be dropped. Servers will often just drop the connection without providing any additional information - you may or may not get a reset by peer error, depending on how the server does the dropping (often, you just get the end event and nothing more - no error raised).