GPU freezes randomly while training using tf.keras models

Question

Versions being used: tensorflow-gpu: 2.0, CUDA v10, CuDNN v7.6.5, Python 3.7.4

System specs: i9-7920X, 4 x RTX 2080Ti, 128GB 2400MHz RAM, 2TB SATA SSD

Issue:

While training any model using tensorflow 2.0, randomly during a epoch, the GPU will freeze and the power usage of the GPU will fall to around 70W with Core usage sitting at 0 and memory utilization also fixed at some random value. I also do not get any error or exception when this happens. Only way to restore is to restart the jupyter kernel and run from the beginning. I first thought that probably something was wrong with my code. So I figured I would try to replicate the issue while training a Densenet on Cifar100 and the issue persisted.

If I run the training on multiple GPUs, then too the GPUs freeze, but it happens very rarely. But with single GPU, it is guaranteed to get stuck at some point or the other.

Below is the code used for training Cifar100

from densenet import DenseNet
from tensorflow.keras.datasets import cifar100
import tensorflow as tf
from tqdm import tqdm_notebook as tqdm

# the data, shuffled and split between train and test sets
(X_train, y_train), (X_test, y_test) = cifar100.load_data(label_mode='fine')
num_classes = 100
y_test_original = y_test

# Convert class vectors to binary class matrices. [one hot encoding]
y_train = tf.keras.utils.to_categorical(y_train, num_classes)
y_test = tf.keras.utils.to_categorical(y_test, num_classes)

X_train = X_train.astype('float32')
X_test = X_test.astype('float32')

for i in range(3):
    mean = np.mean(X_train[:,:,:,i])
    std = np.std(X_train[:,:,:,i])
    X_train[:,:,:,i] = (X_train[:,:,:,i] - mean)/std
    X_test[:,:,:,i] = (X_test[:,:,:,i] - mean)/std


with tf.device('/gpu:0'):  
    model = DenseNet(input_shape=(32,32,3), dense_blocks=3, dense_layers=-1, growth_rate=12, nb_classes=100, dropout_rate=0.2,
             bottleneck=True, compression=0.5, weight_decay=1e-4, depth=100)


optimizer = tf.keras.optimizers.SGD(learning_rate=0.01, 
                                    momentum=0.9, 
                                    nesterov=True,
                                    name='SGD')
model.compile(loss = 'categorical_crossentropy', optimizer = optimizer, metrics = ['accuracy'])

def scheduler(epoch):
    if epoch < 151:
        return 0.01
    elif epoch < 251:
        return 0.001
    elif epoch < 301:
        return 0.0001

callback = tf.keras.callbacks.LearningRateScheduler(scheduler)

model.fit(X_train, y_train, validation_data=(X_test, y_test), epochs=300, verbose = 2)

PS: I even tried the code on my laptop which has a i7-8750h and an RTX 2060 with 32GB and 970 EVO NVME. Unfortunately I had the same problem of GPU freezing.

Does anyone know what the issue is?

Answer 1

So I wanted to post an update on this. The GPU Sync and GPU freeze issues have finally gone. Below are the things I did. I don't know which one fixed it, or if they all contributed towards the fix:

1. DDU GPU drivers and reinstall stock CUDA v10 drivers.
2. Update Motherboard BIOS
3. use the following piece of code.

configproto = tf.compat.v1.ConfigProto() 
configproto.gpu_options.allow_growth = True
sess = tf.compat.v1.Session(config=configproto) 
tf.compat.v1.keras.backend.set_session(sess)