Input of layer sequential is incompatible with the layer: shapes error in LSTM

Question

I'm new to neural networks and I want to use them to compare with other machine learning methods. I have a multivariate time series data with a range of approximately two years. I want to predict 'y' for the next few days based on the other variables using LSTM. The final day of my data is 2020-07-31.

df.tail()

              y   holidays  day_of_month    day_of_week month   quarter
   Date                     
 2020-07-27 32500      0      27                 0        7        3
 2020-07-28 33280      0      28                 1        7        3
 2020-07-29 31110      0      29                 2        7        3
 2020-07-30 37720      0      30                 3        7        3
 2020-07-31 32240      0      31                 4        7        3

To train the LSTM model I also split the data into train and test data.

from sklearn.model_selection import train_test_split
split_date = '2020-07-27' #to predict the next 4 days
df_train = df.loc[df.index <= split_date].copy()
df_test = df.loc[df.index > split_date].copy()
X1=df_train[['day_of_month','day_of_week','month','quarter','holidays']]
y1=df_train['y']
X2=df_test[['day_of_month','day_of_week','month','quarter','holidays']]
y2=df_test['y']

X_train, y_train =X1, y1
X_test, y_test = X2,y2

Because I'm working with LSTM, some scaling is needed:

scaler = MinMaxScaler()
X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)

Now, onto the difficult part: the model.

num_units=50
activation_function = 'sigmoid'
optimizer = 'adam'
loss_function = 'mean_squared_error'
batch_size = 10
num_epochs = 100

 # Initialize the RNN
regressor = Sequential()

 # Adding the input layer and the LSTM layer
regressor.add(LSTM(units = num_units, return_sequences=True ,activation = activation_function, 
input_shape=(X_train.shape[1], 1)))

 # Adding the output layer
regressor.add(Dense(units = 1))

 # Compiling the RNN
regressor.compile(optimizer = optimizer, loss = loss_function)

# Using the training set to train the model
regressor.fit(X_train_scaled, y_train, batch_size = batch_size, epochs = num_epochs)

However, I receive the following error:

ValueError: Input 0 of layer sequential_11 is incompatible with the layer: expected ndim=3, found 
ndim=2. Full shape received: [None, 5]

I don't understand how we choose the parameters or the shape of the input. I've seen some videos and read some Github pages and everyone seems to run LSTM in a different way, which makes it even more difficult to implement. The previous error is probably coming from the shape but other than that is everything else right? And how can I fix this to work? Thanks

EDIT: This similar question does not solve my problem.. I've tried the solution from there

x_train = X_train_scaled.reshape(-1, 1, 5)
x_test  = X_test_scaled.reshape(-1, 1, 5)

(My X_test and y_test only have one column). And the solution also doesn't seem to work. I get this error now:

ValueError: Input 0 is incompatible with layer sequential_22: expected shape= 
(None, None, 1), found shape=[None, 1, 5]

Answer 1

INPUT:

The problem is that you model expect a 3D input of shape (batch, sequence, features) but your X_train is actually a slice of data frame, so a 2D array :

X1=df_train[['day_of_month','day_of_week','month','quarter','holidays']]
X_train, y_train =X1, y1

I assume your columns are supposed to be you features, so what you would usually do is "stack slices" of your df so that you X_train look something like that :

Here is a dummy 2D data set of shape (15,5) :

data = np.zeros((15,5))

array([[0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.],
       [0., 0., 0., 0., 0.]])

You can reshape it to add a batch dimension, for example (15,1,5):

data = data[:,np.newaxis,:] 

array([[[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]],

       [[0., 0., 0., 0., 0.]]])

Same data, but presented in a different way. Now in this example, batch = 15 and sequence = 1, I don't know what is the sequence length in your case but it can be anything.

MODEL :

Now in your model, keras input_shape expect (batch, sequence, features), when you pass this :

input_shape=(X_train.shape[1], 1)

This is what you model sees : (None, Sequence = X_train.shape[1] , num_features = 1) None is for the batch dimension. I don't think that's what your are trying to do so once you've reshaped you should also correct input_shape to match the new array.