Menu
Does anyone know of a library with a working implementation of backpropagation through time? Any of Java/Python/C#/VB.NET/F# (preferably the last one) will do!
680
Backpropagation Through Time, or BPTT, is the application of the Backpropagation training algorithm to recurrent neural network applied to sequence data like a time series. A recurrent neural network is shown one input each timestep and predicts one output. Conceptually, BPTT works by unrolling all input timesteps.
The Backpropagation algorithm is suitable for the feed forward neural network on fixed sized input-output pairs. The Backpropagation Through Time is the application of Backpropagation training algorithm which is applied to the sequence data like the time series. It is applied to the recurrent neural network.
Back-propagation algorithm For l→k, we thus have the time complexity O(lt+lt+ltk+lk)=O(l∗t∗k).
The gradient wrt the hidden state flows backward to the copy node where it meets the gradient from the previous time step. You see, a RNN essentially processes sequences one step at a time, so during backpropagation the gradients flow backward across time steps. This is called backpropagation through time.
Assuming you're already using some library for BP, it should be (TM) rather straightforward to implement BPTT using BP as a step in the process.
The Wikipedia entry for BPTT [1] includes relevant pseudo code.
My own starting point, about 18 years ago, was "The Truck Backer-Upper: An Example of Self-Learning in Neural Networks" [2].
[1] http://en.wikipedia.org/wiki/Backpropagation_through_time
[2] http://www-isl.stanford.edu/~widrow/papers/c1989thetruck.pdf
56
I've used NeuronDotNet only for a limited time though. It allows you to create a feed-forward BackPropagation NN. I especially liked their use of intuitively named classes. Good luck!
This is a .net library.
40
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With
