pm2 managed process in errored state after machine reboot

Question

I'm using pm2 (version 3.4.1) to manage the Laravel Echo websocket server. It runs perfect, but I'm trying to get it to auto-start after a machine reboot (which runs Ubuntu 16.04). To do this I followed these instructions:

$ pm2 startup
[PM2] Init System found: systemd
[PM2] To setup the Startup Script, copy/paste the following command:
sudo env PATH=$PATH:/usr/bin /usr/local/share/.config/yarn/global/node_modules/pm2/bin/pm2 startup systemd -u kramer65 --hp /home/kramer65

$ sudo env PATH=$PATH:/usr/bin /usr/local/share/.config/yarn/global/node_modules/pm2/bin/pm2 startup systemd -u kramer65 --hp /home/kramer65
[sudo] password for kramer65: 
[PM2] Init System found: systemd
Platform systemd
Template
[Unit]
Description=PM2 process manager
Documentation=https://pm2.keymetrics.io/
After=network.target

[Service]
Type=forking
User=kramer65
LimitNOFILE=infinity
LimitNPROC=infinity
LimitCORE=infinity
Environment=PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/bin:/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin
Environment=PM2_HOME=/home/kramer65/.pm2
PIDFile=/home/kramer65/.pm2/pm2.pid

ExecStart=/usr/local/share/.config/yarn/global/node_modules/pm2/bin/pm2 resurrect
ExecReload=/usr/local/share/.config/yarn/global/node_modules/pm2/bin/pm2 reload all
ExecStop=/usr/local/share/.config/yarn/global/node_modules/pm2/bin/pm2 kill

[Install]
WantedBy=multi-user.target

Target path
/etc/systemd/system/pm2-kramer65.service
Command list
[ 'systemctl enable pm2-kramer65' ]
[PM2] Writing init configuration in /etc/systemd/system/pm2-kramer65.service
[PM2] Making script booting at startup...
[PM2] [-] Executing: systemctl enable pm2-kramer65...
[PM2] [v] Command successfully executed.
+---------------------------------------+
[PM2] Freeze a process list on reboot via: $ pm2 save

[PM2] Remove init script via: $ pm2 unstartup systemd

So then I ran pm2 save:

$ pm2 save
[PM2] Saving current process list...
[PM2] Successfully saved in /home/kramer65/.pm2/dump.pm2

After this I reboot the machine, ssh back in, and check pm2 processes:

$ pm2 status
┌──────────┬────┬─────────┬──────┬─────┬─────────┬─────────┬────────┬─────┬────────┬────────┬──────────┐
│ App name │ id │ version │ mode │ pid │ status  │ restart │ uptime │ cpu │ mem    │ user   │ watching │
├──────────┼────┼─────────┼──────┼─────┼─────────┼─────────┼────────┼─────┼────────┼────────┼──────────┤
│ echo     │ 0  │ N/A     │ fork │ N/A │ errored │ 0       │ 0      │ 0%  │ 0 B    │ kramer65 │ disabled │
└──────────┴────┴─────────┴──────┴─────┴─────────┴─────────┴────────┴─────┴────────┴────────┴──────────┘
 Use `pm2 show <id|name>` to get more details about an app

As you can see the status is errored. If I stop and start the process (pm2 stop echo && pm2 start echo) the process runs again.

The error log (~/.pm2/pm2.log) tells me this:

2019-05-14T11:56:36: PM2 log: ===============================================================================
2019-05-14T11:56:36: PM2 log: --- New PM2 Daemon started ----------------------------------------------------
2019-05-14T11:56:36: PM2 log: Time                 : Tue May 14 2019 11:56:36 GMT+0200 (CEST)
2019-05-14T11:56:36: PM2 log: PM2 version          : 3.4.1
2019-05-14T11:56:36: PM2 log: Node.js version      : 8.15.1
2019-05-14T11:56:36: PM2 log: Current arch         : x64
2019-05-14T11:56:36: PM2 log: PM2 home             : /home/kramer65/.pm2
2019-05-14T11:56:36: PM2 log: PM2 PID file         : /home/kramer65/.pm2/pm2.pid
2019-05-14T11:56:36: PM2 log: RPC socket file      : /home/kramer65/.pm2/rpc.sock
2019-05-14T11:56:36: PM2 log: BUS socket file      : /home/kramer65/.pm2/pub.sock
2019-05-14T11:56:36: PM2 log: Application log path : /home/kramer65/.pm2/logs
2019-05-14T11:56:36: PM2 log: Process dump file    : /home/kramer65/.pm2/dump.pm2
2019-05-14T11:56:36: PM2 log: Concurrent actions   : 2
2019-05-14T11:56:36: PM2 log: SIGTERM timeout      : 1600
2019-05-14T11:56:36: PM2 log: ===============================================================================
2019-05-14T11:56:36: PM2 log: App [echo:0] starting in -fork mode-
2019-05-14T11:56:36: PM2 log: App [echo:0] online
2019-05-14T11:56:36: PM2 error: Error: spawn node ENOENT
    at Process.ChildProcess._handle.onexit (internal/child_process.js:190:19)
    at onErrorNT (internal/child_process.js:362:16)
    at _combinedTickCallback (internal/process/next_tick.js:139:11)
    at process._tickDomainCallback (internal/process/next_tick.js:219:9)

Does anybody have any idea what's going wrong here?

Answer 1

ENOENT stands for No such directory entry

What is env <command> doing? (by Chris Johnsen)

The env command is used to run another program in a custom environment without modifying the current one.

sudo env PATH=$PATH:/usr/bin \
   /usr/local/share/.config/yarn/global/node_modules/pm2/bin/pm2 \
   startup systemd -u kramer65 --hp /home/kramer65

Installation note from pm2 documentation To detect available init systems on your machine and generate a configuration, use:

pm2 startup
$ [PM2] You have to run this command as root. Execute the following command:
$ sudo su -c env PATH=$PATH:/home/unitech/.nvm/versions/node/v4.3/bin pm2 startup <distribution> -u <user> --hp <home-path>

I will break down the command as follows

sudo env name=value name2=value2 program and args

1) The program

runs the command program /usr/local/share/.config/yarn/global/node_modules/pm2/bin/pm2

The first remaining argument specifies the program name to invoke; it is searched for according to the `PATH' environment variable.

2) The arguments passed

Any remaining arguments are passed as arguments to that program.

The args passed to pm2 cli command are startup systemd -u kramer65 --hp /home/kramer65.

• the startups command will simply set pm2 to start at boot (Startup Cli Commands)
• the systemd command is used for supported init systems Ubuntu => 16, CenOS >=7, Arch, Debian >=7 (Startup Hooks Compatability) and should be changed to the correct platform as by the included docs.

3) The environment used to execute the program

with an environment formed by extending the current environment with the environment variables and values designated by name=value and name2=value2.

As by the pm2 docs should be a folder inside /home/kramer/.nvm/versions/node/v4.3/bin

PATH=$PATH:/home/unitech/.nvm/versions/node/v4.3/bin

while you are using a folder outside of your home directory because you are running an old version on nodejs (see last chapter Nodejs installed in usr/bin/node).

PATH=$PATH:/usr/bin

4) User permissions

you are passing --user kramer65 --home-path /home/kramer65 to the pm2 cli command.

-u <user> --hp <home-path>

Save your process

The startup hook auto load the process list that you have previously saved. Save your process list with:

pm2 save

Nodejs installed in usr/bin/node (Error: spawn ENOENT on Ubuntu 14.04)

How to reproduce: you have two version of nodejs on your computer. One is an old version installed in /usr/bin/node and the other one is a recent version located in ./nodejs/node. I launch my pm2 manager programmatically, by calling ./nodejs/node myManager.js, which will call pm2.connect(), which will try to spawn the PM2 Daemon.

What happen is that the PM2 Daemon will be launched using the old node binary from /usr/bin/node instead of the recent node binary running my myManager.js script (locatedin ./nodejs/node).

What this does is look for a node binary in the current directory, and then default to node binaries in the path. BUT the node binary running the current code is in ./nodejs/node. To launch the PM2 daemon with the same node binary as the current process, you need to use process.execPath

You can read more info at and in the relevant discussion