Data Shape / Format for RNNs with Multiple Features

Question

I'm trying to build an RNN using python / keras. I understand how it's done with one feature (with t+1 being the output), but how is done with multiple features?

What if I had a regression problem and a dataset with a few different features, one expected output, and I wanted to have the time steps / window set to 30 (so a month if each step represents a day) - what would the shape of the data be? In this example I'd want to be able to predict the output n time periods in the future.

See below for an example of what this data would look like:

I'm having a hard time intuitively understanding the best shape / format the data needs to be for RNNs.

In addition, how well do RNNs handle data sets with, say, 500 features and a few thousand records?

Hopefully someone could help answer or point me in the right direction to get one - so far I've posted on Reddit and Cross Validated with no luck :(

If a code data example is preferred:

# random df
df = pd.DataFrame({'date': np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]),
                   'feature_1': np.random.randint(10, size=10),
                   'feature_2': np.random.randint(10, size=10),
                   'feature_3': np.random.randint(10, size=10),
                   'feature_4': np.random.randint(10, size=10),
                   'output': np.random.randint(10, size=10)}
                 )

# set date as index
df.index = df.date
df = df.drop('date', 1)

Answer 1

Let's say you have 2 timeseries X and Y and you want to predict X using both timeseries. If we choose a timestep of 3 and suppose that we have at our disposal, (X1,...,Xt) and (Y1,...,Yt), the first sample would be : [[X1,X2,X3],[Y1,Y2,Y3]] and the associated output : X4. The second one would be [[X2,X3,X4],[Y2,Y3,Y4]] with X5 as output. And the last one : [[Xt-3,Xt-2,Xt-1],[Yt-3,Yt-2,Yt-1]] with Xt as output.

For example, in the first sample : first you will feed to the network (X1,Y1), then (X2,Y2) and (X3,Y3).

Here is a code to create the input and output and then use a LSTM net to do prediction :

import pandas as pd                                                                                                                                                                                                                                                                
import numpy as np                                                                                                                                                                                                                                                                 
import keras.optimizers                                                                                                                                                                                                                                                            
from keras.models import Sequential                                                                                                                                                                                                                                                
from keras.layers import Dense,Activation                                                                                                                                                                                                                                          
from keras.layers import LSTM                                                                                                                                                                                                                                                      

#random df                                                                                                                                                                                                                                                                        
df = pd.DataFrame({'date': np.array([0, 1, 2, 3, 4, 5, 6, 7, 8, 9]),                                                                                                                                                                                                               
               'feature_1': np.random.randint(10, size=10),                                                                                                                                                                                                                    
               'feature_2': np.random.randint(10, size=10),                                                                                                                                                                                                                    
               'feature_3': np.random.randint(10, size=10),                                                                                                                                                                                                                    
               'feature_4': np.random.randint(10, size=10),                                                                                                                                                                                                                    
               'output': np.random.randint(10, size=10)}                                                                                                                                                                                                                       
             )                                                                                                                                                                                                                                                                 

# set date as index                                                                                                                                                                                                                                                                
df.index = df.date                                                                                                                                                                                                                                                                 
df = df.drop('date', 1)                                                                                                                                                                                                                                                            

nb_epoch = 10                                                                                                                                                                                                                                                                      
batch_size = 10                                                                                                                                                                                                                                                                    
learning_rate = 0.01                                                                                                                                                                                                                                                               
nb_units = 50                                                                                                                                                                                                                                                                       
timeStep = 3                                                                                                                                                                                                                                                                       

X = df[['feature_'+str(i) for i in range(1,5)]].values # Select good columns                                                                                                                                                                                                        
sizeX = X.shape[0]-X.shape[0]%timeStep  # Choose a number of observations that is a multiple of the timstep                                                                                                                                                                            
X = X[:sizeX]                                                                                                                                                                                                                                                                      
X = X.reshape(X.shape[0]/timeStep,timeStep,X.shape[1]) # Create X with shape (nb_sample,timestep,nb_features)                                                                                                                                                                       

Y = df[['output']].values                                                                                                                                                                                                                                                          
Y = Y[range(3,len(Y),3)] #Select the good output                                                                                                                                                                                                                                   


model = Sequential()                                                                                                                                                                                                                                                               
model.add(LSTM(input_dim = X.shape[2],output_dim = nb_units,return_sequences = False)) # One LSTM layer with 50 units                                                                                                                                                               
model.add(Activation("sigmoid"))                                                                                                                                                                                                                                                   
model.add(Dense(1)) #A dense layer which is the final layer                                                                                                                                                                                                                        
model.add(Activation('linear'))                

KerasOptimizer = keras.optimizers.RMSprop(lr=learning_rate, rho=0.9, epsilon=1e-08, decay=0.0)                                                                                                                                                                                     
model.compile(loss="mse", optimizer=KerasOptimizer)                                                                                                                                                                                                                                
model.fit(X,Y,nb_epoch = nb_epoch,batch_size = batch_size)                                                                                                                                                                                                                         
prediction = model.predict(X)