Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

list memory usage in ipython and jupyter

I have a few (almost ten) Gb of memory taken by the ipython kernel. I think this is coming from large objects (matrices, lists, numpy arrays, ...) that I might have produced during some operation and now I do not need anymore.

I would like to list all of the objects I have defined and sort them by their memory footprint. Is there a simple way to do that? For certain types there is nbytes method, but not for all ... so I am looking for a general way to list all objects I have made and their memory occupation.

like image 856
Rho Phi Avatar asked Dec 06 '16 10:12

Rho Phi


People also ask

How do I check memory usage on Jupyter notebook?

The jupyter-resource-usage extension is part of the default installation, and tells you how much memory your user is using right now, and what the memory limit for your user is. It is shown in the top right corner of the notebook interface.

How do I check memory usage in Python?

You can use it by putting the @profile decorator around any function or method and running python -m memory_profiler myscript. You'll see line-by-line memory usage once your script exits.

How much RAM does Jupyter notebook use?

Memory and disk space required per user: 512MB RAM + 1GB of disk + . 5 CPU core. Server overhead: 2-4GB or 10% system overhead (whatever is larger), . 5 CPU cores, 10GB disk space.

How do you clear the RAM on a Jupyter notebook?

Jupyter Lesson 10: How to reset the Kernel (Clear all in memory objects and stop the code) If the notebook or your code is acting weird sometimes it is best to press the “RESET BUTTON”. Reseting the kernel clears all in memory objects and restarts your code from the very top.


1 Answers

Assuming that you are using ipython or jupyter, you will need to do a little bit of work to get a list all of the objects you have defined. That means taking everything available in globals() and filtering out objects that are modules, builtins, ipython objects, etc. Once you are sure you have those objects, then you can proceed to grabbing their sizes with sys.getsizeof. This can be summed up as follows:

import sys  # These are the usual ipython objects, including this one you are creating ipython_vars = ['In', 'Out', 'exit', 'quit', 'get_ipython', 'ipython_vars']  # Get a sorted list of the objects and their sizes sorted([(x, sys.getsizeof(globals().get(x))) for x in dir() if not x.startswith('_') and x not in sys.modules and x not in ipython_vars], key=lambda x: x[1], reverse=True) 

Please keep in mind that for python objects (those created with python's builtin functions), sys.getsizeof will be very accurate. But it can be a bit inaccurate on objects created using third-party libraries. Furthermore, please be mindful that sys.getsizeof adds an additional garbage collector overhead if the object is managed by the garbage collector. So, some things may look a bit heavier than they actually are.

As a side note, numpy's .nbytes method can be somewhat misleading in that it does not include memory consumed by non-element attributes of the array object.

I hope this helps.

like image 169
Abdou Avatar answered Sep 28 '22 11:09

Abdou