NARX implementation using keras

Question

I am trying to implement a simple NARX network in keras (backend Tensoflow). I am building the model by subclassing tf.keras.Model. A NARX network can be trained as a simple feedforward network (series-parallel architecture), therefore I did not have any problem with this definition. However for prediction I need to close the loop and give back the output as input to the model. How can I modify the network so that it is able to make predictions and forecastings?

Here is a minimal version of the code:

import tensorflow as tf
from tensorflow import keras

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
numPreviousSteps = 8
inputShape = (None, numPreviousSteps + 2)

class Narx(keras.Model):

    def __init__(self):
        super(Narx, self).__init__(name='narx')
        self.dense = keras.layers.Dense(10, input_shape=inputShape,
                                        activation=keras.activations.tanh)
        self.outputLayer = keras.layers.Dense(1, activation=keras.activations.linear)

    def call(self, inputs, training = False):
        if (training):
            x = self.dense(inputs)
            return self.outputLayer(x)
        else: # TODO: what should the network do when used for prediction
            x = self.dense(inputs)
            return self.outputLayer(x)


model = Narx()
model.compile(optimizer=keras.training.RMSPropOptimizer(0.001),
              loss=tf.losses.mean_squared_error,
              metric=tf.metrics.mean_absolute_error)

# input data generation
numTsSamples = 1000

# time series to learn from
y = np.random.random((numTsSamples + numPreviousSteps + 1,))
x = np.random.random((numTsSamples,)) # exogenous input

# creation of tapped delay
data = [np.roll(y, -i)[:numTsSamples] for i in range(numPreviousSteps, -1, -1)]
data = [x] + data

# training data
data = np.stack(data, axis=1)

# expected results
yNext = y[numPreviousSteps : -1]

# model training
model.fit(data, yNext)

Answer 1

You can Implement your ARX Model using Gekko: https://apmonitor.com/wiki/index.php/Apps/ARXTimeSeries

I have the same Problem with the feedback Loop. If you are only interested in whether your model can simulate the system behavior, the following code is sufficient. its from: https://aleksandarhaber.github.io/machine_learning/2018/12/20/mlp-sys-identification.html

predict_time=X_test.shape[0]-2*past

Y_predicted_offline=np.zeros(shape=(predict_time,1))
Y_past=network_prediction[0:past,:].T
X_predict_offline=np.zeros(shape=(1,2*past))

for i in range(0,predict_time):
    X_predict_offline[:,0:past]=X_test[i+2*past,0:past]
    X_predict_offline[:,past:2*past]=Y_past    
    y_predict_tmp= model.predict(X_predict_offline)
    Y_predicted_offline[i]=y_predict_tmp
    Y_past[:,0:past-1]=Y_past[:,1:]
    Y_past[:,-1]=y_predict_tmp

error_offline=Y_predicted_offline-Y_test[past:-past,:]
error_offline_percentage=LA.norm(error_offline,2)/LA.norm(Y_test,2)*100

#plot the offline prediction and the real output
plt.plot(Y_test[past:-past,:],'b',label='Real output')
plt.plot(Y_predicted_offline, 'r', label='Offline prediction')
plt.xlabel('Discrete time steps')
plt.ylabel('Output')
plt.legend()    
plt.savefig('prediction_offline.png')
#plt.show()

plt.figure()
#plot the absolute error (offline and online)
plt.plot(abs(error_offline),'r',label='Offline error')
plt.plot(abs(error),'b',label='Online error')
plt.xlabel('Discrete time steps')
plt.ylabel('Absolute prediction error')
plt.yscale('log')
plt.legend()   
plt.savefig('errors.png')
#plt.show()

You can find the explanation in video form there: https://www.youtube.com/watch?v=badm72VMwCE