Predicting sequence of grid coordinates with PyTorch

Question

I have a similar open question here on Cross Validated (though not implementation focused, which I intend this question to be, so I think they are both valid).

I'm working on a project that uses sensors to monitor a persons GPS location. The coordinates will then be converted to a simple-grid representation. What I want to try and do is after recording a users routes, train a neural network to predict the next coordinates, i.e. take the example below where a user repeats only two routes over time, Home->A and Home->B.

I want to train an RNN/LSTM with sequences of varying lengths e.g. (14,3), (13,3), (12,3), (11,3), (10,3), (9,3), (8,3), (7,3), (6,3), (5,3), (4,3), (3,3), (2,3), (1,3) and then also predict with sequences of varying lengths e.g. for this example route if I called

route = [(14,3), (13,3), (12,3), (11,3), (10,3)] //pseudocode
pred = model.predict(route)

pred should give me (9,3) (or ideally even a longer prediction e.g. ((9,3), (8,3), (7,3), (6,3), (5,3), (4,3), (3,3), (2,3), (1,3))

How do I feed such training sequences to the init and forward operations identified below?

self.rnn = nn.RNN(input_size, hidden_dim, n_layers, batch_first=True)
out, hidden = self.rnn(x, hidden)

Also, should the entire route be a tensor or each set of coordinates within the route a tensor?

Answer 1

I'm not very experienced with RNNs, but I'll give it a try.

A few things to pay attention to before we start:
1. Your data is not normalized.
2. The output prediction you want (even after normalization) is not bounded to [-1, 1] range and therefore you cannot have tanh or ReLU activations acting on the output predictions.

To address your problem, I propose a recurrent net that given a current state (2D coordinate) predicts the next state (2D coordinates). Note that since this is a recurrent net, there is also a hidden state associated with each location. At first, the hidden state is zero, but as the net sees more steps, it updates its hidden state.

I propose a simple net to address your problem. It has a single RNN layer with 8 hidden states, and a fully connected layer on to to output the prediction.

class MyRnn(nn.Module):
  def __init__(self, in_d=2, out_d=2, hidden_d=8, num_hidden=1):
    super(MyRnn, self).__init__()
    self.rnn = nn.RNN(input_size=in_d, hidden_size=hidden_d, num_layers=num_hidden)
    self.fc = nn.Linear(hidden_d, out_d)

  def forward(self, x, h0):
    r, h = self.rnn(x, h0)
    y = self.fc(r)  # no activation on the output
    return y, h

You can use your two sequences as training data, each sequence is a tensor of shape Tx1x2 where T is the sequence length, and each entry is two dimensional (x-y).

To predict (during training):

rnn = MyRnn()
pred, out_h = rnn(seq[:-1, ...], torch.zeros(1, 1, 8))  # given time t predict t+1
err = criterion(pred, seq[1:, ...])  # compare prediction to t+1

Once the model is trained, you can show it first k steps and continue to predict the next steps:

rnn.eval()
with torch.no_grad():
  pred, h = rnn(s[:k,...], torch.zeros(1, 1, 8, dtype=torch.float))
  # pred[-1, ...] is the predicted next step
  prev = pred[-1:, ...]
  for j in  range(k+1, s.shape[0]):
    pred, h = rnn(prev, h)  # note how we keep track of the hidden state of the model. it is no longer init to zero.
    prev = pred

I put everything together in a colab notebook so you can play with it.
For simplicity, I ignored the data normalization here, but you can find it in the colab notebook.

---

What's next?
These types of predictions are prone to error accumulation. This should be addressed during training, by shifting the inputs from the ground truth "clean" sequences to the actual predicted sequences, so the model will be able to compensate for its errors.