Not able to reproduce results with Tensorflow even with random seed

Question

I am training a simple autoencoder in Keras with data I have generated. I am currently running the code inside a Google Colab notebook (in case there is a small chance that might be relevant). To achieve reproducible results, I am currently setting random seeds as I have shown below, but it does not seem to be fully effective:

# Choose random seed value 
seed_value = 0

# Set numpy pseudo-random generator at a fixed value
np.random.seed(seed_value)

# Set tensorflow pseudo-random generator at a fixed value
import tensorflow as tf
tf.random.set_seed(seed_value)

The random seed code seems to help with obtaining the same initial weights every time I initialize the model. I can see this using model.get_weights() after creating the model (this is the case even when I restart the notebook and re-run the code). However, I am not able to achieve reproducible results in terms of model performance, because the model weights are different each time after training. I assume that the random seed code above accounts for ensuring the data is split and shuffled in the same way each time during training, even though I have not split the training/validation data beforehand (I am instead using validation_split=0.2) or specified shuffle=False while fitting the model, but perhaps I am incorrect in making that assumption? Additionally, are there any other random seeds I need to include to ensure reproducible results? Here is the code I am using to construct and train the model:

def construct_autoencoder(input_dim, encoded_dim):
   # Add input
   input = Input(shape=(input_dim,))

   # Add encoder layer
   encoder = Dense(encoded_dim, activation='relu')(input)

   # Add decoder layer
   # Input contains binary values, hence the sigmoid activation
   decoder = Dense(input_dim, activation='sigmoid')(encoder)
   model = Model(inputs=input, outputs=decoder)

   return model

autoencoder = construct_autoencoder(10, 6)
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
# print(autoencoder.get_weights()) -> This is the same every time, even with restarting the notebook

autoencoder.fit([data,
                 data, 
                 epochs=20, 
                 validation_split=0.2,
                 batch_size=16,
                 verbose=0)

# print(autoencoder.get_weights()) -> This is different every time, but not sure why?

If you have any ideas on why I am not getting reproducible results during model training, do let me know. I found this https://keras.io/getting-started/faq/#how-can-i-obtain-reproducible-results-using-keras-during-development on Keras' website, but not sure if it's relevant for this (and if so, why?). I know there are other questions asking about reproducibility for model training, but I didn't find any of them to address this specific question. Thank you very much!

Answer 1

In tensorflow 2.x version

this code can reproduce results if you use tf in cpu.

seed_value = 42
import tensorflow as tf
tf.random.set_seed(seed_value)

But if you use tf in gpu (default), the NVIDIA problem issue will make your results unreproducible even you write tf.random.set_seed(seed_value)

So the solution is:

pip install tensorflow-determinism

and then use the following code

def setup_seed(seed):
    random.seed(seed)  
    np.random.seed(seed) 
    tf.random.set_seed(seed)  # tf cpu fix seed
    os.environ['TF_DETERMINISTIC_OPS'] = '1'  # tf gpu fix seed, please `pip install tensorflow-determinism` first


setup_seed(42)