Python os.fork OSError : [Errno 12] Cannot allocate memory (but memory not the issue)

Question

I have similar problem to this one: Python subprocess.Popen "OSError: [Errno 12] Cannot allocate memory"

I have a daemon process that runs OK for a few minutes and then fails to run shell programs via popen2.Popen3(). It spawns 20 threads. Memory does not appear to be the issue; this is the only program running on the machine, which has 2G of RAM, and it's using less than 400M. I've been logging ru_maxrss and this is only 50M (before and after OSError is raised).

ulimit -a:

core file size          (blocks, -c) 0
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 15962
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1024
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) 15962
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

I've also been watching free -m and ls /proc/$pid/fd | wc -l while it is running, and neither of these seems to indicate resource exhaustion. Here's typical free -m while running:

             total       used       free     shared    buffers     cached
Mem:          2003        374       1628          0         46        154
-/+ buffers/cache:        173       1830
Swap:          283          0        283

... and the fd count is around 90-100.

The host is Ubuntu 12.04 (server jeos - minimal vm), Python 2.7.3, running on a VMWare host.

So I'm wondering: what do I do next to diagnose why this is failing? Are there some more resource stats I can gather? Do I need to get down to the level of strace?

Answer 1

Hypothesis: if your VM is 32-bit, you may be running out of address space.

Not memory: address space. Let me explain: in Linux many things (IO, video card, memory-mapped files) use up address space without necessarily consuming corresponding amount of main memory.

Here's an explanation of related issues:

http://us.download.nvidia.com/XFree86/Linux-x86/331.89/README/knownissues.html

(look for "Kernel virtual address space exhaustion on the X86 platform" section, use dmesg to test if that's the situation)

ENOMEM error in result of mmap may very well mean situation of "not enough address space", not just "not enough memory", although I'm not sure how to diagnose this in CPython. If you have some big files mmaped on your system by any process running on it, well..

Answer 2

Check if you have run out of space on your disk drive, that was the problem in my case.

bravo@by1-dotbravo-01:~$ df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/sda1              16G   16G     0 100% /
tmpfs                 2.0G     0  2.0G   0% /dev/shm
/dev/sdb              296G  162G  119G  58% /home