Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Computer restarts with large mini batches in TensorFlow

Tags:

tensorflow

I am running TensorFlow for Windows with a Titan X GPU (12 GB memory). When I try to train a network for images of 256X256X1 with mini-batches larger than 50 images, my computer just crashes and restarts automatically. With smaller mini-batches it runs just fine. Any clues on what might be causing this?

like image 987
Eyal Dekel Avatar asked Dec 20 '16 07:12

Eyal Dekel


1 Answers

I've seen similar problems being discussed in some gaming forums, where the PC would just shut down when the GPU was under heavy load. The reason was usually that the GPU was drawing more power than the power supply unit could handle. Check e.g. here or here. So may be it's worth investigating whether your PSU is the culprit.

Edit: May be the program SpeedFan can help you debugging this - it is able to show both voltages and readings of temperature sensors, which would also tell you if your PC is overheating (I've never used the tool myself, and I'm not affiliated with it either, just found it online).

like image 138
kafman Avatar answered Sep 20 '22 05:09

kafman