Mongo service crashed. Need to find reason of crash

Question

Today morning I found the following error with mongo, on my server.

*** System restart required ***
You have mail.
ubuntu@ip-xxx-xx-xx-xx:~$ mongo
MongoDB shell version: 2.4.9
connecting to: test
Wed Nov 25 05:18:51.381 Error: couldn't connect to server 127.0.0.1:27017 at src/mongo/shell/mongo.js:145
exception: connect failed

It crashed my website. How can I find what stopped mongo service? Do mongo have logs for this? Where can I find them? Also How to make sure if if mongodb is crashed then it restart service as well?

I restarted mongodb service and website started working again.

Output of ps -ef | grep mongod:

mongodb   14756      1  0 05:20 ?        00:00:14 /usr/bin/mongod --config /etc/mongodb.conf
ubuntu    36865  14123  0 06:18 pts/2    00:00:00 grep --color=auto mongod

Answer 1

I had the same issue. After some searching I guessed it was os that sacrificed mongod for memory. If kernel killed your mongod you can find the logs in kernel logs

Mostly kernel logs will be in /var/log/kern.log, The logs might be rotated.

So better to do a grep

grep mongod /var/log/kern.log*

if the logs are archived find -name \*kern.log.*.gz -print0 | xargs -0 zgrep "mongod"

./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118361] mongod invoked oom-killer: gfp_mask=0x24201ca, order=0, oom_score_adj=0
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118365] mongod cpuset=/ mems_allowed=0
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118371] CPU: 0 PID: 1830 Comm: mongod Not tainted 4.4.0-1062-aws #71-Ubuntu
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118531] [ 1301]   112  1301   300787   181207     485       4        0             0 mongod
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.118680] Out of memory: Kill process 1301 (mongod) score 355 or sacrifice child
./kern.log.1:Nov 12 12:55:04 ip-xxx-xxx-xxx-xxx kernel: [8413229.130161] Killed process 1301 (mongod) total-vm:1203148kB, anon-rss:724828kB, file-rss:0kB