How to validate the output given by the Probabilistic Neural Network in MATLAB?

Question

I am interested in learning about the neural networks, and as an example, I tried with the following dataset which has been derived experimentally.

I am using the following input vector for my neural network;

X = [1 1; 1 2; 1 3; 1 4; 4 1; 4 2; 4 3; 4 4; 7 1;7 2; 7 3; 7 4]';
Tc = [1 1 2 3 1 1 2 2 1 1 2 2];

I want to divide the input data into three classes, described by the input vector Tc. Then I am converting the target class indices Tc to vectors T and spread value I am using is 1.

Using newpnn function in MATLAB, I am getting the decision boundaries for the three classes.

I have a doubt regarding the validating if the decision boundary is appropriate. I am validating the output with a single data X =[2;3.55] belongs to class 2. It is depicted by the black dot in the output plot. Blue is class 1. The yellow is the region belonging to class 2. Red is class 3.

As shown in the plot, the prediction by the neural network was found to be class 2, which coincides with the actual class of the set.

So, does that mean that my neural network is correct and validated?

P.S. I have a basic understanding of Neural Networks. Also, I understand the concept of having more training examples and validation sets. I am expecting an answer catering to the available details, as I cannot get more data experimentally.

Answer 1

Hmm, I think that you doesn't understand well was validation is in term of neural networks. You can't check your network only with one sample. So, I'll try to teach you what I know about validating neural networks. It's a long statistical process that involves some reflexion about "real world data", "expected behaviour", ... You can't validate something with like 10-20 data and one validating point.

Generally, when you teach a neural network you should have 3 sets:

• The first one, the training set is the input of the algorithm which is used to set the weights of the different networks. It's just a kind of mandatory data used to run the algorithm.
• The second set, the validation set is used to choose the right algorithm for your problem and to reduce overfitting. It compares the performance of the different ones and to pick the best (the overfitted one won't have a good performance at all).
• The test set: it's the last phase. After having chosen the algorithm and it's parameter, you use a set of new data (taken from the real world) and you check if it performs what it's supposed to do (it's like a coherence test).

(source: https://stats.stackexchange.com/questions/19048/what-is-the-difference-between-test-set-and-validation-set)

For instance, we're building an algorithm used to check if a person "has some chance to become rich". Here's how you would make and valid your neural network.

1. First, we ask 10 000 people if they're rich or not and we check some parameters (age, location, ...). It makes the "original data set".
2. We split that list of 10 000 people into 3 set (6000 2000 and 2000): the training set, the validation set and the test set (note: the proportion can change depending on the validating procedure).
3. We apply the learning set (6000 first data) and we apply it to our different neural networks to teach them (let's name them A, B, C and D)
4. We take the validating set (2000 next data) to check the performance of the four networks. Here's how it avoids overfitting. Let's assume that network A isn't a network at all, it's just a recorder. It records the different datas and their classes but isn't able to predict anything at all. That "dummy algorithm" would have given a 100% result if we make the validation test with the 6000 first person but will completely fail on that test. So, after that test, you can pick the "best algorithm". Let's pick C.
5. Now, we run C with the rest of the data (test set, or new datas if we can, it's always better). If we see that C has a very strange and unpredictable behaviour (it can be caused by some human error such as making sets that aren't really independent or still correct, for instance if the data were from 1996), we pick another algorithm or we try to check what's the problem with the data or the algorithm.

It's how you can make a reliable neural network (don't forget, the two main problems are don't checking the final result and overfitting).

As overfitting is a key notion. I'll try to define it a bit and to give an example. Overfitting is making an algorithm that is able to build very close approximations but that aren't able to predict anything (what I called "dummy algorithm").

Let's compare for instance a linear interpolator and a polynomial (1000000th degree, very high degree) one. Our polynomial algorithm will probably fit very well the data (extreme overfitting is fitting exactly all our data). But, it won't be able to predict anything at all.

For the example below, if we have in our validation set (extracted from real world data) a point in (2,-2) and (-1,2), we can assume that our polynomial interpolation was clearly overfitted because it suggests values such as (-1,10) and (2,20). The linear one should be closer.

I hope it will help. (note that I'm not an expert in that domain but I tried to make a very readable and simple answer, so if anything is false, feel free to comment :) )