Disk quota exceeded on OpenShift

Question

I just deployed application to OpenShift, and it worked fine, until I ment to push changes. Then it started to warn me about disk quota exceeded, and I'm not able to do anything anymore.

Running rhc app-tidy [app-name] from terminal gave me this:

Warning: Gear 5395736e5004467cae0004ba is using 100.0% of disk quota
Failed to execute: 'control start' for /var/lib/openshift/5395736e5004467cae0004ba/nodejs

app-root/logs folder is empty.

Running du -h * | sort -rh | head -50, gives me this:

481M    mongodb/data
481M    mongodb
385M    mongodb/data/journal
275M    app-root/runtime
275M    app-root
267M    app-deployments/2014-06-09_04-51-12.537
267M    app-deployments
157M    app-root/runtime/repo
156M    app-deployments/2014-06-09_04-51-12.537/repo
125M    app-root/runtime/repo/node_modules
125M    app-deployments/2014-06-09_04-51-12.537/repo/node_modules
119M    app-root/runtime/dependencies/.npm
119M    app-root/runtime/dependencies
111M    app-deployments/2014-06-09_04-51-12.537/dependencies/.npm
111M    app-deployments/2014-06-09_04-51-12.537/dependencies

Not sure why mongodb/data is taking so much space and what exactly is there? My database should be empty.

** update **

After running on my mongodb/data folder, here's what I got:

total 97M
drwxr-xr-x.  3 5395736e5004467cae0004ba 5395736e5004467cae0004ba 4.0K Jun  9 05:21 .
drwxr-xr-x. 11 5395736e5004467cae0004ba 5395736e5004467cae0004ba 4.0K Jun  9 04:44 ..
-rw-------.  1 5395736e5004467cae0004ba 5395736e5004467cae0004ba  16M Jun  9 04:45 admin.0
-rw-------.  1 5395736e5004467cae0004ba 5395736e5004467cae0004ba  16M Jun  9 04:45 admin.ns
drwxr-xr-x.  2 5395736e5004467cae0004ba 5395736e5004467cae0004ba 4.0K Jun  9 06:51 journal
-rw-------.  1 5395736e5004467cae0004ba 5395736e5004467cae0004ba  16M Jun  9 06:51 local.0
-rw-------.  1 5395736e5004467cae0004ba 5395736e5004467cae0004ba  16M Jun  9 06:51 local.ns
-rwxr-xr-x.  1 5395736e5004467cae0004ba 5395736e5004467cae0004ba    0 Jun  9 06:51 mongod.lock
-rw-------.  1 5395736e5004467cae0004ba 5395736e5004467cae0004ba  16M Jun  9 04:45 surge.0
-rw-------.  1 5395736e5004467cae0004ba 5395736e5004467cae0004ba  16M Jun  9 04:45 surge.ns

Answer 1

Something that also helped me was to disable journaling

Laying out complete steps

1. SSH into your OpenShift

   rhc ssh $@ -a $app
   

2. In the MongoDB configuration

   vim ~/mongodb/conf/mongodb.conf
   

   add or set

   nojournal = true
   

Note: ~ automatically resolves to /var/lib/openshift/<YOUR_OPENSHIFT_ID>

3. Delete previous journal data

   rm ~/mongodb/data/journal/*
   

Then exit from SSH, and

4. restart MongoDB

   rhc cartridge-restart mongodb-2.4 -a $app
   

This of course, comes at the cost of not journaling, which is helpful for recovering the data in case of corruption or unclean shutdown. But if you're just testing things out and running out of space then disabling journaling can be very helpful as it takes a lot of space.

Answer 2

If you're hitting the disk quota that's either because you hit the size or inode limit.

To fix the size you can clean up tmp folders, app logs, git repos etc. You can use:

rhc app-tidy -a <appname>

To check if you're hitting the inode limit use quota command:

quota -s

To fix the quota issue, you will need to keep number of files to a minimum.

If that doesn't help and you can try to shutdown the MongoDB and delete the contents of mongodb folder (i.e. use helmy's advice).

Answer 3

MongoDB preallocates data files and may preallocate journal files.

If your database is empty and/or you don't care about the data, try stopping mongod and then blowing away your data directory. Then verify/set the following mongod options:

smallfiles=true
noprealloc=true