Sagemaker: MemoryError: Unable to allocate ___for an array with shape ___ and data type float64

Question

I am running a notebook in sagemaker and it seems like one of the arrays produced after vectorizing text is causing issues.

Reading other answers it seems like it is an issue with overcommit. And one of the solutions proposed is to set it to always overcommit with this:

$ echo 1 > /proc/sys/vm/overcommit_memory

Is there any documentation or do you have any suggestion on how to do the same thing in sagemaker?

Thank you very much.

Answer 1

Open a root shell by sudo -i and then run the echo 1 > /proc/sys/vm/overcommit_memory