TensorFlow exited abnormally with code 137

Question

I'm trying convolutional neural net using TensorFlow.

Though I could success some training, the script failed with

Process Python exited abnormally with code 137

when I just changed the training data. The data sizes of the first and second data are the same, and I could train CNN using the very second data on another platform (lasagne). (The first data is slice of photo images, and the second is artificially generated geological patterns).

Can someone tell me the meaning of the error or the way to avoid this problem?

My platform is Mac OS X 10.10.5, Python 2.7.10 |Anaconda 2.2.0 (x86_64), and tensorflow-0.5.0.

Thank you in advance.

Answer 1

Exit code 137 means that your Python process was killed by the SIGKILL signal. It's hard to say for certain, but one possibility is that your process was killed by the OOM (out-of-memory) killer. Check /var/log/messages to see if there is any information about why your process was killed.

This post has advice on how to disable the OOM killer for an individual process. However, this ignores the root cause, which is that your process (or TensorFlow itself) is using too much memory.