Pytorch RuntimeError: CUDA out of memory with a huge amount of free memory

Question

While training the model, I encountered the following problem:

RuntimeError: CUDA out of memory. Tried to allocate 304.00 MiB (GPU 0; 8.00 GiB total capacity; 142.76 MiB already allocated; 6.32 GiB free; 158.00 MiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation. See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF

As we can see, the error occurs when trying to allocate 304 MiB of memory, while 6.32 GiB is free! What is the problem? As I can see, the suggested option is to set max_split_size_mb to avoid fragmentation. Will it help and how to do it correctly?

This is my version of PyTorch:

torch==1.10.2+cu113     
torchvision==0.11.3+cu113     
torchaudio===0.10.2+cu113

Answer 1

I wasted several hours until I discovered that reducing the batch size and resizing the width of my input image (image size) were necessary steps.

Answer 2

Your problem may be due to fragmentation of your GPU memory.You may want to empty your cached memory used by caching allocator.

import torch
torch.cuda.empty_cache()