Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

How to free up all memory pytorch is taken from gpu memory

I have some kind of high level code, so model training and etc. are wrapped by pipeline_network class. My main goal is to train new model every new fold.

for train_idx, valid_idx in cv.split(meta_train[DEPTH_COLUMN].values.reshape(-1)):

        meta_train_split, meta_valid_split = meta_train.iloc[train_idx], meta_train.iloc[valid_idx]

        pipeline_network = unet(config=CONFIG, suffix = 'fold' + str(fold), train_mode=True)

But then I move on to 2nd fold everything fails out of gpu memory:

RuntimeError: cuda runtime error (2) : out of memory at /pytorch/torch/lib/THC/generic/THCStorage.cu:58

At the end of epoch I tried to manually delete that pipeline with no luck:

 def clean_object_from_memory(obj): #definition
    del obj
    gc.collect()
    torch.cuda.empty_cache()

clean_object_from_memory( clean_object_from_memory) # calling

Calling this didn't help as well:

def dump_tensors(gpu_only=True):
        torch.cuda.empty_cache()
        total_size = 0
        for obj in gc.get_objects():
            try:
                if torch.is_tensor(obj):
                    if not gpu_only or obj.is_cuda:
                        del obj
                        gc.collect()
                elif hasattr(obj, "data") and torch.is_tensor(obj.data):
                    if not gpu_only or obj.is_cuda:
                        del obj
                        gc.collect()
            except Exception as e:
                pass

How can reset pytorch then I move on to the next fold?

like image 253
Rocketq Avatar asked Sep 06 '18 13:09

Rocketq


People also ask

How do I release CUDA memory in PyTorch?

To release the memory, you would have to make sure that all references to the tensor are deleted and call torch. cuda. empty_cache() afterwards. E.g. del bottoms should only delete the internal bottoms tensor, while the global one should still be alive.

How do I remove models from GPU PyTorch?

Like said above: if you want to free the memory on the GPU you need to get rid of all references pointing on the GPU object. Then it will be freed automatically. So assuming model is on GPU: model=model. cpu() will free the GPU-memory if you don't keep any other references to of model, but model_cpu=model.

How do I check my PyTorch GPU usage?

The easiest way to check if you have access to GPUs is to call torch. cuda. is_available(). If it returns True, it means the system has the Nvidia driver correctly installed.


1 Answers

Try delete the object with del and then apply torch.cuda.empty_cache(). The reusable memory will be freed after this operation.

like image 112
HzCheng Avatar answered Oct 11 '22 12:10

HzCheng