ubuntu "No space left on device" but there is tons of space [closed]

Question

I am getting the "No space left on device" error for pretty much anything i try to do. Even using tab to autocomplete a command!

but when i do df -h i get:

ubuntu@ip-10-0-2-108:~$ df -h Filesystem      Size  Used Avail Use% Mounted on /dev/xvda1       99G  6.5G   88G   7% / udev            3.7G  8.0K  3.7G   1% /dev tmpfs           1.5G  184K  1.5G   1% /run none            5.0M     0  5.0M   0% /run/lock none            3.7G     0  3.7G   0% /run/shm none            100M     0  100M   0% /run/user /dev/xvdb       414G  199M  393G   1% /mnt overflow        1.0M  1.0M     0 100% /tmp 

which to me looks like there is tons of space. df -i also looks similar:

ubuntu@ip-10-0-2-108:~$ df -i Filesystem       Inodes IUsed    IFree IUse% Mounted on /dev/xvda1      6553600 94227  6459373    2% / udev             951353   393   950960    1% /dev tmpfs            953649   274   953375    1% /run none             953649     3   953646    1% /run/lock none             953649     1   953648    1% /run/shm none             953649     1   953648    1% /run/user /dev/xvdb      27525120    11 27525109    1% /mnt overflow         953649    12   953637    1% /tmp 

I am on an Amazon EC2 ubuntu 12.04 instance.

Here are some examples of the error popping up:

ubuntu@ip-10-0-2-108:~$ sudo crontab -e /tmp/crontab.RvYjrR/crontab: No space left on device  ubuntu@ip-10-0-2-108:~$ ls /va (hit tab for autocomplete)  -bash: cannot create temp file for here-document: No space left on device -bash: cannot create temp file for here-document: No space left on device 

however the server seems to be running and everything seems to be working. what on earth is going on??

Answer 1

It's possible that you've run out of memory or some space elsewhere and it prompted the system to mount an overflow filesystem, and for whatever reason, it's not going away.

Try unmounting the overflow partition:

umount /tmp 

or

umount overflow