How to debug dying Jupyter Python3 kernel?

Question

I'm running some code using scipy and scikits.learn on Jupyter notebook using Python 3 kernel. During the computation the kernel is being restarted with a message dialogue saying that “The kernel appears to have died. It will restart automatically.”. The stderr of the underlying Jupyter process just logs the fact that the kernel dies and is going to be restarted without any helpful message. Is there any way of checking the underlying error? It might be a segfault coming from within some C++ code, but I can only guess. I searched for any relevant logs on the server and failed to find anything helpful.

Answer 1

Faced the exact same issue while reading close to 5000 images as a numpy array in an 8 gigs RAM laptop, for a Machine learning project. After doing a bit of math with the resolution of my images, the size of a respective numpy array, I figured that 8 gigs of RAM is not sufficient to handle the images. After a lot of research on the net, which involved suggestions like updating CUDA, cuDNN, downgrading TensorFlow (they faced the same error while importing the relevant modules/packages), update numpy to the latest version and update the intel Math Kernel Version (command: "conda install -c intel mkl")(a whole day's research). The solution that worked for me was to run the model training process on Google colab.

Now, getting back to your question: The displayed dialogue: “The kernel appears to have died. It will restart automatically.” is not an "error" per se. It is more like "Jupyter Notebook helping itself" by clearing out all the variables and restarting the kernel. It is Jupyter Notebook sending an SOS signal, and getting help from itself so that it does not crash. Which otherwise would cause the restarted Jupyter Notebook to not have the unsaved changes. (Well, it autosaves, but does not "auto checkpoint")

This "response" of Jupyter Notebook is simply because the maximum RAM capacity of your laptop is reached. - This is the "underlying error"(response). This will deallocate the resources, enabling you to restart the program. Recall your computer hanging when you open too many tabs of chrome? or run a program that has too many variables' values to be stored (like in my 5000 images case)? This could have been the alternative response of Jupyter Notebook when the RAM capacity is fully utilized. Hanging. Or crashing.

But instead, the developers have been kind enough to enable it to take care of itself.

Note1: Running the same code as .py script, errors will be more verbose.

Note2: If you are using CUDA, remember that Jupyter Notebook fails to deallocate CUDA resources even when the session is terminated. So this might be the reason for it to restart.