mongodb crashes after unexpected shutdown

Question

after our vps got down and we boot it, mongod service didnt start automatically and we cannot start it by service mongod start, file /var/log/mongo/mongod.log contains following log:

  ***** SERVER RESTARTED *****


Thu Oct 11 13:51:18 
Thu Oct 11 13:51:18 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Thu Oct 11 13:51:18 
Thu Oct 11 13:51:18 [initandlisten] MongoDB starting : pid=3821 port=27017 dbpath=/var/lib/mongo 32-bit host=server2.paransa.org
Thu Oct 11 13:51:18 [initandlisten] 
Thu Oct 11 13:51:18 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Thu Oct 11 13:51:18 [initandlisten] **       see http://blog.mongodb.org/post/137788967/32-bit-limitations
Thu Oct 11 13:51:18 [initandlisten] **       with --journal, the limit is lower
Thu Oct 11 13:51:18 [initandlisten] 
Thu Oct 11 13:51:18 [initandlisten] ** WARNING: You are running in OpenVZ. This is known to be broken!!!
Thu Oct 11 13:51:18 [initandlisten] 
Thu Oct 11 13:51:18 [initandlisten] db version v2.2.0, pdfile version 4.5
Thu Oct 11 13:51:18 [initandlisten] git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207
Thu Oct 11 13:51:18 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49
Thu Oct 11 13:51:18 [initandlisten] options: { config: "/etc/mongod.conf", dbpath: "/var/lib/mongo", fork: "true", logappend: "true", logpath: "/var/log/mongo/mongod.log", pidfilepath: "/var/run/mongodb/mongod.pid" }
Thu Oct 11 13:51:18 [initandlisten] couldn't open /var/lib/mongo/mydb_main.ns errno:13 Permission denied
Thu Oct 11 13:51:18 [initandlisten] error couldn't open file /var/lib/mongo/mydb_main.ns terminating
Thu Oct 11 13:51:18 dbexit: 
Thu Oct 11 13:51:18 [initandlisten] shutdown: going to close listening sockets...
Thu Oct 11 13:51:18 [initandlisten] shutdown: going to flush diaglog...
Thu Oct 11 13:51:18 [initandlisten] shutdown: going to close sockets...
Thu Oct 11 13:51:18 [initandlisten] shutdown: waiting for fs preallocator...
Thu Oct 11 13:51:18 [initandlisten] shutdown: closing all files...
Thu Oct 11 13:51:18 [initandlisten] closeAllFiles() finished
Thu Oct 11 13:51:18 [initandlisten] shutdown: removing fs lock...
Thu Oct 11 13:51:18 dbexit: really exiting now

as you may see, log says mongo server cant open *.ns file cause lake of permission

but if i run the server manually by following command:

mongod --dbpath=/var/lib/mongo -f /etc/mongod.conf

it seems everything is fine and about the log file:

***** SERVER RESTARTED *****


Thu Oct 11 14:02:06 
Thu Oct 11 14:02:06 warning: 32-bit servers don't have journaling enabled by default. Please use --journal if you want durability.
Thu Oct 11 14:02:06 
Thu Oct 11 14:02:06 [initandlisten] MongoDB starting : pid=4090 port=27017 dbpath=/var/lib/mongo 32-bit host=server2.paransa.org
Thu Oct 11 14:02:06 [initandlisten] 
Thu Oct 11 14:02:06 [initandlisten] ** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
Thu Oct 11 14:02:06 [initandlisten] **       see http://blog.mongodb.org/post/137788967/32-bit-limitations
Thu Oct 11 14:02:06 [initandlisten] **       with --journal, the limit is lower
Thu Oct 11 14:02:06 [initandlisten] 
Thu Oct 11 14:02:06 [initandlisten] ** WARNING: You are running in OpenVZ. This is known to be broken!!!
Thu Oct 11 14:02:06 [initandlisten] 
Thu Oct 11 14:02:06 [initandlisten] db version v2.2.0, pdfile version 4.5
Thu Oct 11 14:02:06 [initandlisten] git version: f5e83eae9cfbec7fb7a071321928f00d1b0c5207
Thu Oct 11 14:02:06 [initandlisten] build info: Linux domU-12-31-39-01-70-B4 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_49
Thu Oct 11 14:02:06 [initandlisten] options: { config: "/etc/mongod.conf", dbpath: "/var/lib/mongo", fork: "true", logappend: "true", logpath: "/var/log/mongo/mongod.log", pidfilepath: "/var/run/mongodb/mongod.pid" }
Thu Oct 11 14:02:07 [websvr] admin web console waiting for connections on port 28017
Thu Oct 11 14:02:07 [initandlisten] waiting for connections on port 27017

and using --repair arg didnt help (maybe because it could start manually with mongod --dbpath=/var/lib/mongo -f /etc/mongod.conf and adding --repair arg to the command, logs everything is fine!...)

any suggestion?

Answer 1

This is a permissions issue, you are running as mongod (or mongodb etc.) user when you start via service, and you are likely running as root (you confirm this in the comments above) when you are using sudo or invoking as root. You need to fix the permissions for all the files in that folder and not start the service as root, ever.

To fix the files based on what you have posted, assuming your relevant user is mongodb and the group is mongodb, you would do the following with sudo (or as root) - substitute for the appropriate user for your system:

chown -R mongodb:mongodb /var/lib/mongo
chown -R mongodb:mongodb /var/run/mongodb
chown mongodb:mongodb /var/log/mongo/mongod.log

Once you fix the permissions on these folders you should be able to start via service again. Note: if you post the output of ls -al /var/lib/mongo then I can give you the exact commands you need to run