nginx crashing daily and error.log shows nothing

Question

I have an nginx server setup as a reverse proxy that seems to be crashing daily. There was never any issue with the server before but recently (a month or so ago) I started noticing that nginx wasn't running and I'd have to log into the server to start the process up again.

I'm having trouble finding anything useful in the logs. I'd appreciate any help in diagnosing the issue.

nginx version: nginx/1.10.3 (Ubuntu)

os: Ubuntu 16.04.4 LTS (running in an LXC)

# systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Sat 2018-06-23 21:49:46 UTC; 1min 23s ago
Process: 13485 ExecStop=/sbin/start-stop-daemon --quiet --stop --retry QUIT/5 --pidfile /run/nginx.pid (code=exited, status=1/FAILURE)
Process: 13402 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Process: 13401 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
Main PID: 13403 (code=exited, status=0/SUCCESS)

Jun 23 10:30:17 nginx systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 23 10:30:17 nginx systemd[1]: Started A high performance web server and a reverse proxy server.

cat /var/log/nginx/error.log

2018/06/23 21:49:46 [notice] 13484#13484: signal process started

There was nothing suspicious in the access.log file.

Let me know if there's any more information that would be helpful

Answer 1

I had the same problem and it was the same source of error: certbot was turning off the nginx server and was not capable of starting it again after renewal.

PROBLEM:

You can check if you encounter the same problem by checking following logs. First nginx logs:

tail -n 100 /var/log/nginx/error.log

Result:

2019/02/05 12:07:37 [notice] 1629#1629: signal process started
2019/02/05 12:07:37 [error] 1629#1629: open() "/run/nginx.pid" failed (2: No such file or directory)
2019/02/05 12:07:38 [emerg] 1655#1655: bind() to 0.0.0.0:80 failed (98: Address already in use)
2019/02/05 12:07:38 [emerg] 1655#1655: bind() to 0.0.0.0:443 failed (98: Address already in use)
2019/02/05 12:07:38 [emerg] 1655#1655: bind() to [::]:443 failed (98: Address already in use)
2019/02/05 12:07:38 [emerg] 1655#1655: bind() to 0.0.0.0:444 failed (98: Address already in use)
2019/02/05 12:07:38 [emerg] 1655#1655: bind() to [::]:444 failed (98: Address already in use)
[...]
2019/02/05 12:07:38 [emerg] 1655#1655: still could not bind()
2019/02/05 12:07:41 [alert] 1631#1631: unlink() "/run/nginx.pid" failed (2: No such file or directory)

We see nginx unsuccessfully trying to restart.

You can chek syslog too:

tail -n 100 /var/log/syslog

And look for the same timestamp:

Feb  5 12:07:30 systemd[1]: Starting Certbot...
Feb  5 12:07:31 systemd[1]: Stopping A high performance web server and a reverse proxy server...
Feb  5 12:07:31 systemd[1]: Stopped A high performance web server and a reverse proxy server.
Feb  5 12:07:38 systemd[1]: Starting A high performance web server and a reverse proxy server...

We see that certbot seems to cause the problem.

SOLUTION:

In my case, I had an old version of certbot. You can check your version using certbot --version command. In my case I had certbot 0.10.2...

So first of all, upgrade your certbot application, and add nginx plugin:

sudo apt-get update
sudo apt-get install certbot python-certbot-nginx

Check your new version: certbot --version -> certbot 0.28.0.

Then, you will have to modify renewal configuration files accordingly to new version, and using nginx plugin. Renewal conf file are in /etc/letsencrypt/renewal/* directory. Note that the certbot documentation discourage you to manually modify them...

I modify all the renewal configuration files from:

# renew_before_expiry = 30 days
version = 0.10.2
archive_dir = /etc/letsencrypt/archive/yourdomain
cert = /etc/letsencrypt/live/yourdomain/cert.pem
privkey = /etc/letsencrypt/live/yourdomain/privkey.pem
chain = /etc/letsencrypt/live/yourdomain/chain.pem
fullchain = /etc/letsencrypt/live/yourdomain/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = standalone
post_hook = service nginx start
account = yourkey
pre_hook = service nginx stop
installer = nginx

To:

# renew_before_expiry = 30 days
version = 0.28.0
archive_dir = /etc/letsencrypt/archive/yourdomain
cert = /etc/letsencrypt/live/yourdomain/cert.pem
privkey = /etc/letsencrypt/live/yourdomain/privkey.pem
chain = /etc/letsencrypt/live/yourdomain/chain.pem
fullchain = /etc/letsencrypt/live/yourdomain/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = yourkey
server = https://acme-v02.api.letsencrypt.org/directory
authenticator = nginx
installer = nginx

(note that only the version and authenticator lines have been modified, server line has been added, and pre_hook and post_hook lines have been removed).

Then you can check if your next renewal will run smoothly, by simulating a renewal, using the following command:

certbot renew --dry-run

You should get the following for each of your certificate, with no red error:

new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/yourdomain/fullchain.pem