Menu
I am new to tensorflow and I am trying to train on the CIFAR 10 dataset. I noticed that no matter what batch size I use according to my nvidia control panel I am using 97% of my gpu memory. I tried batch sizes of 100 down to 2 and in each case my gpu memory usage is always 97%. Why would it do this?
def batchGenerator(batch_size = 32):
bi = 0
random.shuffle(train_data)
while bi + batch_size < len(train_data):
x = np.zeros((batch_size, 32, 32, 3))
y = np.zeros((batch_size, 10))
for b in range(batch_size):
x[b] = train_data[bi + b][0]
if random.choice((True, False)):
img = cv2.flip(x[b], 0)
if random.choice((True, False)):
img = cv2.flip(x[b], 1)
y[b][train_data[bi + b][1]] = 1
bi += batch_size
yield(x, y)
with tf.Session() as s:
s.run(tf.initialize_all_variables())
for epoch in range(100):
a = 0.0
i = 0
for x_train, y_train in batchGenerator(2):
outs = s.run([opt, accuracy], feed_dict = {x: x_train, y_exp: y_train, p_keep: 0.5})
a += outs[-1]
i += 1
print('Epoch', epoch, 'Accuracy', a / i)
620
It is now clearly noticeable that increasing the batch size will directly result in increasing the required GPU memory. In many cases, not having enough GPU memory prevents us from increasing the batch size.
Limiting GPU memory growth To limit TensorFlow to a specific set of GPUs, use the tf. config. set_visible_devices method. In some cases it is desirable for the process to only allocate a subset of the available memory, or to only grow the memory usage as is needed by the process.
In any case, no, there isn't anything that Tensorflow can use. Since GPUs don't use "shared graphics memory", the term "shared memory" in relation to GPUs is used for the on-chip cache memory of streaming multiprocessors (devblogs.nvidia.com/using-shared-memory-cuda-cc), which led to some confusion in my answer..
On the opposite, big batch size can really speed up your training, and even have better generalization performances. A good way to know which batch size would be good, is by using the Simple Noise Scale metric introduced in “ An Empirical Model of Large-Batch Training”.
This issue is related to How to prevent tensorflow from allocating the totality of a GPU memory?
TensorFlow uses by default all the memory of your GPU, and this is the normal behavior. From the tutorial Using GPUs:
By default, TensorFlow maps nearly all of the GPU memory of all GPUs visible to the process. This is done to more efficiently use the relatively precious GPU memory resources on the devices by reducing memory fragmentation.
They also provide different options if you need to reduce the memory taken by TensorFlow, such as:
config = tf.ConfigProto()
config.gpu_options.per_process_gpu_memory_fraction = 0.4
session = tf.Session(config=config, ...)
(from the linked docs), to limit the memory use to 40% of the GPU's available memory.
53
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
