Is there a workaround for running out of memory on GPU with tensorflow?

Question

I am currently building a 3d convolutional network, for video classification. The main problem is that I run out of memory too easily. Even if I set my batch_size to 1, there is still not enough memory to train my CNN the way I want.

I am using a GTX 970 with 4Gb of VRAM (3.2Gb free to use by tensorflow). I was expecting it to still train my network, maybe using my RAM memory as a backup, or doing the calculations in parts. But until now I could only run it making the CNN simpler, which affects performance directly.

I think I can run on CPU, but it is significantly slower, making it not a good solution either.

Is there a better solution than to buy a better GPU?

Thanks in advance.

Answer 1

Using gradient checkpointing will help with memory limits.