Memory limit in jupyter notebook

Question

How do I set a maximum memory limit for a jupyter notebook process?

If I use too much RAM the computer gets blocked and I have to press the power button to restart the computer manually.

Is there a way of automatically killing a jupyter notebook process as soon as a user-set memory limit is surpassed or to throw a memory error? Thanks

Answer 1

Under Linux you can use "cgroups" to limit resources for any software running on your computer.

Install cgroup-tools with apt-get install cgroup-tools

Edit its configuration /etc/cgconfig.conf to make a profile for the particular type of work (e.g. numerical scientific computations):

group app/numwork {
  memory {
    memory.limit_in_bytes = 500000000;
  }
}

Apply that configuration to the process names you care about by listing them in /etc/cgrules.conf (in my case it is all julia executables, which I run through jupyter, but you can use it for any other software too):

*:julia  memory  app/numwork/

Finally, parse the config and set it as the current active config with the following commands:

~# cgconfigparser -l /etc/cgconfig.conf
~# cgrulesengd

I use this to set limits on processes running on a server that is used by my whole class of students.

This page has some more details and other ways to use cgroups https://wiki.archlinux.org/index.php/cgroups