Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Looping through training data in Neural Networks Backpropagation Algorithm

How many times do I use a sample of training data in one training cycle? Say I have 60 training data. I go through the 1st row and do a forward pass and adjust weights using results from backward pass. Using the sigmoidal function as below:

Forward pass 
Si = sum of (Wi * Uj)
Ui = f(Si) = 1 / 1 + e^ - Si

Backward pass 
Output Cell = (expected -Ui)(f'(Si)), where 
f'(Si) = Ui(1-Ui)

Do I then go through the 2nd row and do the same process as the 1st or do I go around the 1st row until the error is less?

I hope someone can help please

like image 260
obsessiveCookie Avatar asked Mar 24 '14 16:03

obsessiveCookie


People also ask

How is the training algorithm performed in backpropagation neural network?

The algorithm is used to effectively train a neural network through a method called chain rule. In simple terms, after each forward pass through a network, backpropagation performs a backward pass while adjusting the model's parameters (weights and biases).

What is back propagation training algorithm?

Backpropagation, or backward propagation of errors, is an algorithm that is designed to test for errors working back from output nodes to input nodes. It is an important mathematical tool for improving the accuracy of predictions in data mining and machine learning.

Which rule is used in backpropagation algorithm?

The backpropagation algorithm is based on generalizing the Widrow-Hoff learning rule. It uses supervised learning, which means that the algorithm is provided with examples of the inputs and outputs that the network should compute, and then the error is calculated.


1 Answers

Training the network

You should use each instance of the training set once per training epoch.

A training epoch is a complete cycle through your dataset.

After you've looped through the dataset and calculated the deltas, you should adjust the weights of the network. Then you may perform a new forward pass on the neural network and do another training epoch, looping through your training dataset.

Graphical representation
A really great graphical representation of backpropagation may be found at this link.


Single-step training

There are two approaches to train you network to perform classification on a dataset. The easiest method is called single-step or online learning. This is the method you will find in most litterature, and it is also the fastest to converge. As you train your network you will calculate the deltas for each layer and adjust the weights for each instance of your dataset.

Thus if you have a dataset of 60 instances, this means you should have adjusted the weights 60 times before the training epoch is over.

Batch training

The other approach is called batch training or offline learning. This approach often yields a network with a lower residual error. When you train the network you should calculate the deltas for each layer for every instance of the dataset, and then finally average the individual deltas and correct the weights once per epoch.

If you have a dataset of 60 instances, this means you should have adjusted the weights once before the training epoch is over.

like image 193
jorgenkg Avatar answered Sep 24 '22 15:09

jorgenkg