Logo Questions Linux Laravel Mysql Ubuntu Git Menu
 

Neural Network Cost Function in MATLAB

How would I implement this neural network cost function in matlab:

Neural Network Cost Function

Here are what the symbols represent:

% m is the number of training examples.   [a scalar number]
% K is the number of output nodes.   [a scalar number]
% Y is the matrix of training outputs.   [an m by k matrix]
% y^{(i)}_{k} is the ith training output (target) for the kth output node.   [a scalar number]
% x^{(i)} is the ith training input.   [a column vector for all the input nodes]
% h_{\theta}(x^{(i)})_{k} is the value of the hypothesis at output k, with weights theta, and training input i.   [a scalar number]

%note: h_{\theta}(x^{(i)}) will be a column vector with K rows.

I'm having problems with the nested sums, the bias nodes, and the general complexity of this equation. I'm also struggling because there are 2 matrices of weights, one connecting the inputs to the hidden layer, and one connecting the hidden layer to the outputs. Here's my attempt so far.

Define variables

m = 100            %number of training examples
K = 2              %number of output nodes
E = 2              %number of input nodes
A = 2              %number of nodes in each hidden layer
L = 1              %number of hidden layers

Y = [2.2,   3.5    %targets for y1 and y2 (see picture at bottom of page)
     1.7,   2.1
     1.9,   3.6
      .     .      %this is filled out in the actual code but to save space I have used ellipsis. there will be m rows.
      .     .
      .     .
     2.8,   1.6]

X = [1.1,   1.8    %training inputs. there will be m rows
     8.5,   1.0
     9.5,   1.8
      .     .
      .     .
      .     . 
     1.4,   0.8]

W1 = [1.3,  .    .  0.4    %this is just an E by A matrix of random numbers. this is the matrix of initial weights.
       .    .    .  - 2
       .    .    .  3.1
       .    .    .  - 1
      2.1, -8, 1.2, 2.1]

W2 = [1.3,  .    .  0.4    %this is an A by K matrix of random numbers. this is the matrix of initial weights.
       .    .    .  - 2
       .    .    .  3.1
       .    .    .  - 1
      2.1, -8, 1.2, 2.1]

Hypothesis using these weights equals...

Htheta = sigmf( dot(W2 , sigmf(dot(W1 , X))) )   %This will be a column vector with K rows.

Cost Function using these weights equals... (This is where I am struggling)

  sum1 = 0
  for i = 1:K
  sum1 = sum1 + Y(k,i) *log(Htheta(k)) + (1 - Y(k,i))*log(1-Htheta(k))

I just keep writing things like this and then realising it's all wrong. I can not for the life of me work out how to do the nested sums, or include the input matrix, or do any of it. It's all very complicated.

How would I create this equation in matlab?

Thank you very much!

A 2 layer neural network with 2 inputs, 2 outputs, 2 hidden nodes, and 2 bias units

Note: The code has strange colours as stackoverflow doesn't know I am programing in MATLAB. I have also wrote the code straight into stackoverflow, so it may have syntax errors. I am more interested in the general idea of how I should go about doing this rather than just having a code to copy and paste. This is the reason I haven't bothered with semi colons and such.

like image 661
Blue7 Avatar asked Jan 29 '14 20:01

Blue7


People also ask

What is the sum of cost function of a neural network?

Cost function of a neural network is a generalization of the cost function of the logistic regression. Here the summation term (sum_{k=1}^K) is to generalize over the K output units of the neural network by calculating the cost function and summing over all the output units in the network.

Is it easy to run neural networks in MATLAB?

Running neural networks in matlab is quite understandable once you understand the equations. This is part 5 in my series on neural networks. You are welcome to start at part 1. Here are the previous articles explaining the cost function. Source code used here can be downloaded from github.

What is a neural network in machine learning?

A neural network is a machine learning algorithm that takes in multiple inputs, runs them through an algorithm, and essentially sums the output of the different algorithms to get the final output. The cost function of a neural network will be the sum of errors in each layer.

Is it possible to implement neural networks in MATLAB using octave?

I've implemented neural networks using the same error function as the one you've mentioned above. Unfortunately, I haven't worked with Matlab for quite some time, but I'm fairly proficient in Octave,which hopefully you can still find useful, since many of the functions in Octave are similar to those of Matlab.


2 Answers

I've implemented neural networks using the same error function as the one you've mentioned above. Unfortunately, I haven't worked with Matlab for quite some time, but I'm fairly proficient in Octave,which hopefully you can still find useful, since many of the functions in Octave are similar to those of Matlab.

@sashkello provided a good snippet of code for computing the cost function. However, this code is written with a loop structure, and I would like to offer a vectorized implementation.

In order to evaluate the current theta values, we need to perform a feed forward/ forward propagation throughout the network. I'm assuming you know how to write the feed forward code, since you're only concerned with the J(theta) errors. Let the vector representing the results of your forward propagation be F

Once you've performed feedforward, you'll need to carry out the equation. Note, I'm implementing this in a vectorized manner.

J = (-1/m) * sum(sum(Y .* log(F) + (1-Y) .* log(1-F),2));

This will compute the part of the summation concerning:

part 1 of the total cost

Now we must add the regularization term, which is:

Typically, we would have arbitrary number of theta matrices, but in this case we have 2, so we can just perform several sums to get:

J =J + (lambda/(2*m)) * (sum(sum(theta_1(:,2:end).^2,2)) + sum(sum(theta_2(:,2:end).^2,2)));

Notice how in each sum I'm only working from the second column through the rest. This is because the first column will correspond to the theta values we trained for the `bias units.

So there's a vectorized implementation of the computation of J.

I hope this helps!

like image 85
Alejandro Avatar answered Sep 17 '22 17:09

Alejandro


I think Htheta is a K*2 array. Note that you need to add bias (x0 and a0) in the forward cost function calculation. I showed you the array dimensions in each step under the assumption that you have two nodes at input , hidden, and output layers as comments in the code.

m = size(X, 1);  
X = [ones(m,1) X]; % m*3 in your case
% W1 2*3, W2 3*2
a2 = sigmf(W1 * X');  % 2*m
a2 = [ones(m,1) a2'];  % m*3    
Htheta = sigmf(a2 * W2);  % m*2    

J = (1/m) * sum ( sum (  (-Y) .* log(Htheta)  -  (1-Y) .* log(1-Htheta) ));

t1 = W1(:,2:size(W1,2));
W2 = W2';
t2 = W2(:,2:size(W2,2));

% regularization formula
Reg = lambda  * (sum( sum ( t1.^ 2 )) + sum( sum ( t2.^ 2 ))) / (2*m);
like image 38
lennon310 Avatar answered Sep 19 '22 17:09

lennon310