How would I implement this neural network cost function in matlab:
Here are what the symbols represent:
% m is the number of training examples. [a scalar number]
% K is the number of output nodes. [a scalar number]
% Y is the matrix of training outputs. [an m by k matrix]
% y^{(i)}_{k} is the ith training output (target) for the kth output node. [a scalar number]
% x^{(i)} is the ith training input. [a column vector for all the input nodes]
% h_{\theta}(x^{(i)})_{k} is the value of the hypothesis at output k, with weights theta, and training input i. [a scalar number]
%note: h_{\theta}(x^{(i)}) will be a column vector with K rows.
I'm having problems with the nested sums, the bias nodes, and the general complexity of this equation. I'm also struggling because there are 2 matrices of weights, one connecting the inputs to the hidden layer, and one connecting the hidden layer to the outputs. Here's my attempt so far.
Define variables
m = 100 %number of training examples
K = 2 %number of output nodes
E = 2 %number of input nodes
A = 2 %number of nodes in each hidden layer
L = 1 %number of hidden layers
Y = [2.2, 3.5 %targets for y1 and y2 (see picture at bottom of page)
1.7, 2.1
1.9, 3.6
. . %this is filled out in the actual code but to save space I have used ellipsis. there will be m rows.
. .
. .
2.8, 1.6]
X = [1.1, 1.8 %training inputs. there will be m rows
8.5, 1.0
9.5, 1.8
. .
. .
. .
1.4, 0.8]
W1 = [1.3, . . 0.4 %this is just an E by A matrix of random numbers. this is the matrix of initial weights.
. . . - 2
. . . 3.1
. . . - 1
2.1, -8, 1.2, 2.1]
W2 = [1.3, . . 0.4 %this is an A by K matrix of random numbers. this is the matrix of initial weights.
. . . - 2
. . . 3.1
. . . - 1
2.1, -8, 1.2, 2.1]
Hypothesis using these weights equals...
Htheta = sigmf( dot(W2 , sigmf(dot(W1 , X))) ) %This will be a column vector with K rows.
Cost Function using these weights equals... (This is where I am struggling)
sum1 = 0
for i = 1:K
sum1 = sum1 + Y(k,i) *log(Htheta(k)) + (1 - Y(k,i))*log(1-Htheta(k))
I just keep writing things like this and then realising it's all wrong. I can not for the life of me work out how to do the nested sums, or include the input matrix, or do any of it. It's all very complicated.
How would I create this equation in matlab?
Thank you very much!
Note: The code has strange colours as stackoverflow doesn't know I am programing in MATLAB. I have also wrote the code straight into stackoverflow, so it may have syntax errors. I am more interested in the general idea of how I should go about doing this rather than just having a code to copy and paste. This is the reason I haven't bothered with semi colons and such.
Cost function of a neural network is a generalization of the cost function of the logistic regression. Here the summation term (sum_{k=1}^K) is to generalize over the K output units of the neural network by calculating the cost function and summing over all the output units in the network.
Running neural networks in matlab is quite understandable once you understand the equations. This is part 5 in my series on neural networks. You are welcome to start at part 1. Here are the previous articles explaining the cost function. Source code used here can be downloaded from github.
A neural network is a machine learning algorithm that takes in multiple inputs, runs them through an algorithm, and essentially sums the output of the different algorithms to get the final output. The cost function of a neural network will be the sum of errors in each layer.
I've implemented neural networks using the same error function as the one you've mentioned above. Unfortunately, I haven't worked with Matlab for quite some time, but I'm fairly proficient in Octave,which hopefully you can still find useful, since many of the functions in Octave are similar to those of Matlab.
I've implemented neural networks using the same error function as the one you've mentioned above. Unfortunately, I haven't worked with Matlab for quite some time, but I'm fairly proficient in Octave,which hopefully you can still find useful, since many of the functions in Octave are similar to those of Matlab.
@sashkello provided a good snippet of code for computing the cost function. However, this code is written with a loop structure, and I would like to offer a vectorized implementation.
In order to evaluate the current theta values, we need to perform a feed forward/ forward propagation
throughout the network. I'm assuming you know how to write the feed forward code, since you're only concerned with the J(theta)
errors. Let the vector representing the results of your forward propagation be F
Once you've performed feedforward, you'll need to carry out the equation. Note, I'm implementing this in a vectorized manner.
J = (-1/m) * sum(sum(Y .* log(F) + (1-Y) .* log(1-F),2));
This will compute the part of the summation concerning:
Now we must add the regularization term, which is:
Typically, we would have arbitrary number of theta matrices, but in this case we have 2, so we can just perform several sums to get:
J =J + (lambda/(2*m)) * (sum(sum(theta_1(:,2:end).^2,2)) + sum(sum(theta_2(:,2:end).^2,2)));
Notice how in each sum I'm only working from the second column through the rest. This is because the first column will correspond to the theta
values we trained for the `bias units.
So there's a vectorized implementation of the computation of J.
I hope this helps!
I think Htheta
is a K*2 array. Note that you need to add bias (x0
and a0
) in the forward cost function calculation. I showed you the array dimensions in each step under the assumption that you have two nodes at input , hidden, and output layers as comments in the code.
m = size(X, 1);
X = [ones(m,1) X]; % m*3 in your case
% W1 2*3, W2 3*2
a2 = sigmf(W1 * X'); % 2*m
a2 = [ones(m,1) a2']; % m*3
Htheta = sigmf(a2 * W2); % m*2
J = (1/m) * sum ( sum ( (-Y) .* log(Htheta) - (1-Y) .* log(1-Htheta) ));
t1 = W1(:,2:size(W1,2));
W2 = W2';
t2 = W2(:,2:size(W2,2));
% regularization formula
Reg = lambda * (sum( sum ( t1.^ 2 )) + sum( sum ( t2.^ 2 ))) / (2*m);
If you love us? You can donate to us via Paypal or buy me a coffee so we can maintain and grow! Thank you!
Donate Us With