Keras LSTM - why different results with "same" model & same weights?

Question

(NOTE: Properly fixing the RNG state before each model creating as described in comment in comment practically fixed my problem, as within 3 decimals results are consistent, but they aren't exactly so, so there's somewhere a hidden source of randomness not fixed by seeding the RNG... probably some lib uses time milisecs or smth...if anyone has an idea on that, it would be cool to know, so I will wait and not close question yet :) )

I create a Keras LSTM model (used to predict some time series data, not important what), and every time I try to re-create an identical model (same mode config loaded from json, same weights loaded from file, same args to compile function), I get wildly different results on same train and test data. WHY?

Code is roughly like this:

# fix random
import random
random.seed(42)

# make model & compile
model = Sequential([
    LSTM(50, input_shape=(None, 1), return_sequences=True),
    LSTM(100, return_sequences=False),
    Dense(1),
    Activation("linear")
])
model.compile(loss="mse", optimizer="rmsprop")

# save it and its initial random weights
model_json = model.to_json()
model.save_weights("model.h5")

# fit and predict
model.fit(x_train, y_train, epochs=3)
r = model.predict(x_test)

# create new "identical" model
model2 = model_from_json(model_json)
model2.load_weights("model.h5")
model2.compile(loss="mse", optimizer="rmsprop")

# fit and predict "identical" model
model2.fit(x_train, y_train, epochs=3)
r2 = model2.predict(x_test)

# ...different results :(

I know that the model has initial random weights, so I'm saving them up and reloading them. I'm also paranoid enough to assume there are some "hidden" params that I may not know of, so I serialize model to json and reload instead of recreating an identical one by hand (tried that, same thing btw). And I also fixed the random number generator.

It's my first time with Keras, and I'm also a beginners to neural networks in general. But this this drives me crazy... wtf can vary?!

---

On fixing random number generators: I run Keras with the TensorFlow backend, and I have these lines of code at the start to try and fix the RNGs for experimental purposes:

import random
random.seed(42)
import numpy
numpy.random.seed(42)
from tensorflow import set_random_seed
set_random_seed(42)

...but they still don't fix the randomness.

And I understand that the goal is to make my model to behave non-randomly despite the inherent stochastic nature of NNs. But I need to temporarily fix this for experimental purposes (I'm even OK with it being reproducible on one machine only!).

Answer 1

Machine learning algorithms in general are non-deterministic. This means that every time you run them the outcome should vary. This has to do with the random initialization of the weights. If you want to make the results reproducible you have to eliminate the randomness from the table. A simple way to do this is to use a random seed.

import numpy as np
import tensorflow as tf

np.random.seed(1234)
tf.random.set_seed(1234)

# rest of your code

If you want the randomness factor but not so high variance in your output, I would suggest either lowering your learning rate or changing your optimizer (I would suggest an SGD optimizer with a relatively low learning rate). A cool overview of gradient descent optimization is available here!

---

A note on TensorFlow's random generators is that besides a global seed (i.e. tf.random.set_seed()), they also use an internal counter, so if you run

tf.random.set_seed(1234)
print(tf.random.uniform([1]).numpy())
print(tf.random.uniform([1]).numpy())

You'll get 0.5380393 and 0.3253647, respectively. However if you re-run that same snippet, you'll get the same two numbers again.

A detailed explanation of how random seeds work in TensorFlow can be found here.

---

For newer TF versions take care of this too: TensorFlow 2.2 ships with a os environment variable TF_DETERMINISTIC_OPS which if set to '1', will ensure that only deterministic GPU ops are used.

Answer 2

This code is for keras using tensorflow backend

This is because the weights are initialised using random numbers and hence you will get different results every time. This is expected behaviour. To have reproducible result you need to set the random seed as. Below example set operation-level and graph-level seeds for more information look here

import tensorflow as tf
import random as rn

os.environ['PYTHONHASHSEED'] = '0'

# Setting the seed for numpy-generated random numbers
np.random.seed(37)

# Setting the seed for python random numbers
rn.seed(1254)

# Setting the graph-level random seed.
tf.set_random_seed(89)

from keras import backend as K

session_conf = tf.ConfigProto(
      intra_op_parallelism_threads=1,
      inter_op_parallelism_threads=1)

#Force Tensorflow to use a single thread
sess = tf.Session(graph=tf.get_default_graph(), config=session_conf)

K.set_session(sess)

# Rest of the code follows from here on ...

Answer 3

I resolved this issue by adding os.environ['TF_DETERMINISTIC_OPS'] = '1'

Here an example:

import os
os.environ['TF_DETERMINISTIC_OPS'] = '1'
#rest of the code
# tf version 2.3.1