Low Volatile GPU Utilization when running tensor flow

Question

I'm having trouble making tensorflow to use the Nvidia GeForce GTX 1080 GPU on my system efficiently. I reduced my code to the very simple version shown below; I'm only looping through a session.run() operation that should use the GPU, the data is only fetched once and is reused inside the loop, so this code should only utilize the GPU.

input_training_data=self.val_data[batch_size, :] input_training_label=self.val_label[batch_size, :] feed_dict = self.get_feed_dict(input_training_data, input_training_label) for i in range(1000): acc = sess.run(cost, feed_dict)

I noticed that for batch_size = 16, I get a mostly steadily GPU usage at around 8%, as I increase the batch_size to 32 the maximum GPU usage increases to 9-12% but the utilization stays mostly at 0% and from time to time it jumps to 15%-25% and immediately falls back to 0%. This patterns continues for larger batch_sizes, basically any batch size larger than 16 increases the maximum utilization but the utilization stays mostly at 0 and only spikes up from time to time. What am I missing here?

Answer 1

I've had the same issue. My issue was: My calculation was partially executed on GPU and CPU (so there was a lot of communication between both devices, which lead to a low GPU utilization) I've read in a different thread:

1. Don't use the data_dictornaries in loops (use Tensorvariable instead)
2. There has been an issue with float64 datatype (which HAS to be calculated on CPU only) -> Use float32 datatype (if possible)
3. use the Profiler suggested by Olivier Moindrot to check, if anything is explicitly run on your CPU. Then try to bring everything to GPU

By the way, a hint regarding your code: Your session (default graph) will grow each iteration until a OutOfMemory exception possible... -> I close the session every x-iteration while resetting the default graph...