How can I solve 'ran out of gpu memory' in TensorFlow

Question

I was encountering out of memory errors when training a small CNN on a GTX 970. Through somewhat of a fluke, I discovered that telling TensorFlow to allocate memory on the GPU as needed (instead of up front) resolved all my issues. This can be accomplished using the following Python code:

    config = tf.ConfigProto()
    config.gpu_options.allow_growth = True
    sess = tf.Session(config=config)

Previously, TensorFlow would pre-allocate ~90% of GPU memory. For some unknown reason, this would later result in out-of-memory errors even though the model could fit entirely in GPU memory. By using the above code, I no longer have OOM errors.

Note: If the model is too big to fit in GPU memory, this probably won't help!

It's not about that. first of all you can see how much memory it gets when it runs by monitoring your gpu. for example if you have a nvidia gpu u can check that with watch -n 1 nvidia-smi command. But in most cases if you didn't set the maximum fraction of gpu memory, it allocates almost the whole free memory. your problem is lack of enough memory for your gpu. cnn networks are totally heavy. When you are trying to feed your network DO NOT do it with your whole data. DO this feeding procedure in low batch sizes.

By default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to CUDA_VISIBLE_DEVICES) visible to the process. This is done to more efficiently use the relatively precious GPU memory resources on the devices by reducing memory fragmentation.

TensorFlow provides two Config options on the Session to control this.

The first is the allow_growth option, which attempts to allocate only as much GPU memory based on runtime allocations:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.Session(config=config)

The second method is the per_process_gpu_memory_fraction option, which determines the fraction of the overall amount of memory that each visible GPU should be allocated. For example, you can tell TensorFlow to only allocate 40% of the total memory of each GPU by:

config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.4
session = tf.Session(config=config)

Tensorflow 2

As we don't have sessions anymore the solution is not longer viable.

By default, TensorFlow maps nearly all of the GPU memory of all GPUs (subject to CUDA_VISIBLE_DEVICES) visible to the process.
In some cases, it is desirable for the process to only allocate a subset of the available memory, or to only grow the memory usage as is needed by the process. TensorFlow provides two methods to control this. One of them is using set_memory_growth tf.config.experimental.set_memory_growth

For a full understanding, I recommend this link: Limiting GPU memory growth

For Tensorflow 2:

config = tf.compat.v1.ConfigProto()
config.gpu_options.allow_growth = True
session = tf.compat.v1.Session(config=config)