Menu
I am training PyTorch deep learning models on a Jupyter-Lab notebook, using CUDA on a Tesla K80 GPU to train. While doing training iterations, the 12 GB of GPU memory are used. I finish training by saving the model checkpoint, but want to continue using the notebook for further analysis (analyze intermediate results, etc.).
However, these 12 GB continue being occupied (as seen from nvtop) after finishing training. I would like to free up this memory so that I can use it for other notebooks.
My solution so far is to restart this notebook's kernel, but that is not solving my issue because I can't continue using the same notebook and its respective output computed so far.
973
PyTorch uses a memory cache to avoid malloc/free calls and tries to reuse the memory, if possible, as described in the docs. To release memory from the cache so that other processes can use it, you could call torch. cuda. empty_cache() .
To release the memory, you would have to make sure that all references to the tensor are deleted and call torch. cuda. empty_cache() afterwards. E.g. del bottoms should only delete the internal bottoms tensor, while the global one should still be alive.
The answers so far are correct for the Cuda side of things, but there's also an issue on the ipython side of things.
When you have an error in a notebook environment, the ipython shell stores the traceback of the exception so you can access the error state with %debug. The issue is that this requires holding all variables that caused the error to be held in memory, and they aren't reclaimed by methods like gc.collect(). Basically all your variables get stuck and the memory is leaked.
Usually, causing a new exception will free up the state of the old exception. So trying something like 1/0 may help. However things can get weird with Cuda variables and sometimes there's no way to clear your GPU memory without restarting the kernel.
For more detail see these references:
https://github.com/ipython/ipython/pull/11572
How to save traceback / sys.exc_info() values in a variable?
132
If you just set object that uses a lot of memory to None like this:
obj = None
And after that you call
gc.collect() # Python thing
This is how you may avoid restarting the notebook.
If you still would like to see it clear from Nvidea smi or nvtop you may run:
torch.cuda.empty_cache() # PyTorch thing
to empty the PyTorch cache.
28
with pytorch.no_grad():
torch.cuda.empty_cache()
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
