NeuronDotNet: why does my function return different outputs to the in-built one?

Question

I am using NeuronDotNet for neural networks in C#. In order to test the network (as well as train it), I wrote my own function to get the sum squared error. However, when I tested this function by running it on the training data and comparing it to the MeanSquaredError of the Backpropagation network, the results were different.

I discovered the reason for the different error is that the network is returning different outputs when I run to when its run in the learning phase. I run it for each TrainingSample using:

double[] output = xorNetwork.Run(sample.InputVector);

In the learning phase its using:

xorNetwork.Learn(trainingSet, cycles);

...with a delegate to trap the end sample event:

xorNetwork.EndSampleEvent +=
    delegate(object network, TrainingSampleEventArgs args)
    {
        double[] test = xorNetwork.OutputLayer.GetOutput();
        debug.addSampleOutput(test);
    };

I tried doing this using the XOR problem, to keep it simple, and the outputs are still different. For example, at the end of the first epoch, the outputs from the EndSampleEvent delegate vs those from my function are:

• Input: 01, Expected: 1, my_function: 0.703332, EndSampleEvent 0.734385
• Input: 00, Expected: 0, my_function: 0.632568, EndSampleEvent 0.649198
• Input: 10, Expected: 1, my_function: 0.650141, EndSampleEvent 0.710484
• Input: 11, Expected: 0, my_function: 0.715175, EndSampleEvent 0.647102
• Error: my_function: 0.280508, EndSampleEvent 0.291236

Its not something as simple as it being captured at a different phase in the epoch, the outputs are not identical to those in the next/previous epoch.

I've tried debugging, but I am not an expert in Visual Studio and I'm struggling a bit with this. My project references the NeuronDotNet DLL. When I put breakpoints into my code, it won't step into the code from the DLL. I've looked elsewhere for advice on this and tried several solutions and got nowhere.

I don't think its due to the 'observer effect', i.e. the Run method in my function causing the network to change. I have examined the code (in the project that makes the DLL) and I don't think Run changes any of the weights. The errors from my function tend to be lower than those from the EndSampleEvent by a factor which exceeds the decrease in error from a typical epoch, i.e. its as if the network is getting ahead of itself (in terms of training) temporarily during my code.

Neural networks are stochastic in the sense that they adjust their functions during training. However, the output should be deterministic. Why is it that the two sets of outputs are different?

EDIT: Here is the code I am using.

/***********************************************************************************************
COPYRIGHT 2008 Vijeth D

This file is part of NeuronDotNet XOR Sample.
(Project Website : http://neurondotnet.freehostia.com)

NeuronDotNet is a free software. You can redistribute it and/or modify it under the terms of
the GNU General Public License as published by the Free Software Foundation, either version 3
of the License, or (at your option) any later version.

NeuronDotNet is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY;
without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with NeuronDotNet.
If not, see <http://www.gnu.org/licenses/>.

***********************************************************************************************/

using System;
using System.Collections.Generic;
using System.Drawing;
using System.IO;
using System.Text;
using System.Windows.Forms;
using NeuronDotNet.Core;
using NeuronDotNet.Core.Backpropagation;
using ZedGraph;

namespace NeuronDotNet.Samples.XorSample
{
    public partial class MainForm : Form
    {
        private BackpropagationNetwork xorNetwork;
        private double[] errorList;
        private int cycles = 5000;
        private int neuronCount = 3;
        private double learningRate = 0.25d;

        public MainForm()
        {
            InitializeComponent();
        }

        private void Train(object sender, EventArgs e)
        {
            EnableControls(false);
            if (!int.TryParse(txtCycles.Text.Trim(), out cycles)) { cycles = 5000; }
            if (!double.TryParse(txtLearningRate.Text.Trim(), out learningRate)) { learningRate = 0.25d; }
            if (!int.TryParse(txtNeuronCount.Text.Trim(), out neuronCount)) { neuronCount = 3; }

            if (cycles < 1) { cycles = 1; }
            if (learningRate < 0.01) { learningRate = 0.01; }
            if (neuronCount < 1) { neuronCount = 1; }

            txtNeuronCount.Text = neuronCount.ToString();
            txtCycles.Text = cycles.ToString();
            txtLearningRate.Text = learningRate.ToString();

            errorList = new double[cycles];
            InitGraph();

            LinearLayer inputLayer = new LinearLayer(2);
            SigmoidLayer hiddenLayer = new SigmoidLayer(neuronCount);
            SigmoidLayer outputLayer = new SigmoidLayer(1);
            new BackpropagationConnector(inputLayer, hiddenLayer);
            new BackpropagationConnector(hiddenLayer, outputLayer);
            xorNetwork = new BackpropagationNetwork(inputLayer, outputLayer);
            xorNetwork.SetLearningRate(learningRate);

            TrainingSet trainingSet = new TrainingSet(2, 1);
            trainingSet.Add(new TrainingSample(new double[2] { 0d, 0d }, new double[1] { 0d }));
            trainingSet.Add(new TrainingSample(new double[2] { 0d, 1d }, new double[1] { 1d }));
            trainingSet.Add(new TrainingSample(new double[2] { 1d, 0d }, new double[1] { 1d }));
            trainingSet.Add(new TrainingSample(new double[2] { 1d, 1d }, new double[1] { 0d }));
           Console.WriteLine("mse_begin,mse_end,output,outputs,myerror");
            double max = 0d;
         Console.WriteLine(NNDebug.Header);
           List < NNDebug > debugList = new List<NNDebug>();
           NNDebug debug = null;
         xorNetwork.BeginEpochEvent +=
              delegate(object network, TrainingEpochEventArgs args)
                 {
                  debug = new NNDebug(trainingSet);
                 };

           xorNetwork.EndSampleEvent +=
            delegate(object network, TrainingSampleEventArgs args)
                 {                                                  
                  double[] test = xorNetwork.OutputLayer.GetOutput();

                  debug.addSampleOutput(args.TrainingSample, test);
                 };

         xorNetwork.EndEpochEvent +=
            delegate(object network, TrainingEpochEventArgs args)
            {    
               errorList[args.TrainingIteration] = xorNetwork.MeanSquaredError;
               debug.setMSE(xorNetwork.MeanSquaredError);
               double[] test = xorNetwork.OutputLayer.GetOutput();
               GetError(trainingSet, debug);
               max = Math.Max(max, xorNetwork.MeanSquaredError);
               progressBar.Value = (int)(args.TrainingIteration * 100d / cycles);
               //Console.WriteLine(debug);
               debugList.Add(debug);
            };

            xorNetwork.Learn(trainingSet, cycles);
            double[] indices = new double[cycles];
            for (int i = 0; i < cycles; i++) { indices[i] = i; }

            lblTrainErrorVal.Text = xorNetwork.MeanSquaredError.ToString("0.000000");

            LineItem errorCurve = new LineItem("Error Dynamics", indices, errorList, Color.Tomato, SymbolType.None, 1.5f);
            errorGraph.GraphPane.YAxis.Scale.Max = max;
            errorGraph.GraphPane.CurveList.Add(errorCurve);
            errorGraph.Invalidate();
         writeOut(debugList);
            EnableControls(true);
        }

       private const String pathFileName = "C:\\Temp\\NDN_Debug_Output.txt";

      private void writeOut(IEnumerable<NNDebug> data)
      {
         using (StreamWriter streamWriter = new StreamWriter(pathFileName))
         {
            streamWriter.WriteLine(NNDebug.Header);

            //write results to a file for each load combination
            foreach (NNDebug debug in data)
            {
               streamWriter.WriteLine(debug);
            }
         } 
      }

      private void GetError(TrainingSet trainingSet, NNDebug debug)
      {
         double total = 0;
         foreach (TrainingSample sample in trainingSet.TrainingSamples)
         {
            double[] output = xorNetwork.Run(sample.InputVector);

            double[] expected = sample.OutputVector;
            debug.addOutput(sample, output);
            int len = output.Length;
            for (int i = 0; i < len; i++)
            {
               double error = output[i] - expected[i];
               total += (error * error);
            }
         }
         total = total / trainingSet.TrainingSampleCount;
         debug.setMyError(total);
      }

      private class NNDebug
      {
         public const String Header = "output(00->0),output(01->1),output(10->1),output(11->0),mse,my_output(00->0),my_output(01->1),my_output(10->1),my_output(11->0),my_error";

         public double MyErrorAtEndOfEpoch;
         public double MeanSquaredError;
         public double[][] OutputAtEndOfEpoch;
         public double[][] SampleOutput;
         private readonly List<TrainingSample> samples;

         public NNDebug(TrainingSet trainingSet)
         {
            samples =new List<TrainingSample>(trainingSet.TrainingSamples);
            SampleOutput = new double[samples.Count][];
            OutputAtEndOfEpoch = new double[samples.Count][];
         } 

         public void addSampleOutput(TrainingSample mySample, double[] output)
         {
            int index = samples.IndexOf(mySample);
            SampleOutput[index] = output;
         }

         public void addOutput(TrainingSample mySample, double[] output)
         {
            int index = samples.IndexOf(mySample);
            OutputAtEndOfEpoch[index] = output;
         }

         public void setMyError(double error)
         {
            MyErrorAtEndOfEpoch = error;
         }

         public void setMSE(double mse)
         {
            this.MeanSquaredError = mse;
         }

         public override string ToString()
         {
            StringBuilder sb = new StringBuilder();
            foreach (double[] arr in SampleOutput)
            {
               writeOut(arr, sb);
               sb.Append(',');
            }
            sb.Append(Math.Round(MeanSquaredError,6));
            sb.Append(',');
            foreach (double[] arr in OutputAtEndOfEpoch)
            {
               writeOut(arr, sb);
               sb.Append(',');
            }
            sb.Append(Math.Round(MyErrorAtEndOfEpoch,6));
            return sb.ToString();
         }
      }

      private static void writeOut(double[] arr, StringBuilder sb)
      {
         bool first = true;
         foreach (double d in arr)
         {
            if (first)
            {
               first = false;
            }
            else
            {
               sb.Append(',');
            }
            sb.Append(Math.Round(d, 6));
         }  
      }   

        private void EnableControls(bool enabled)
        {
            btnTrain.Enabled = enabled;
            txtCycles.Enabled = enabled;
            txtNeuronCount.Enabled = enabled;
            txtLearningRate.Enabled = enabled;
            progressBar.Value = 0;
            btnTest.Enabled = enabled;
            txtTestInput.Enabled = enabled;
        }

        private void LoadForm(object sender, EventArgs e)
        {
            InitGraph();
            txtCycles.Text = cycles.ToString();
            txtLearningRate.Text = learningRate.ToString();
            txtNeuronCount.Text = neuronCount.ToString();
        }

        private void InitGraph()
        {
            GraphPane pane = errorGraph.GraphPane;
            pane.Chart.Fill = new Fill(Color.AntiqueWhite, Color.Honeydew, -45F);
            pane.Title.Text = "Back Propagation Training - Error Graph";
            pane.XAxis.Title.Text = "Training Iteration";
            pane.YAxis.Title.Text = "Sum Squared Error";
            pane.XAxis.MajorGrid.IsVisible = true;
            pane.YAxis.MajorGrid.IsVisible = true;
            pane.YAxis.MajorGrid.Color = Color.LightGray;
            pane.XAxis.MajorGrid.Color = Color.LightGray;
            pane.XAxis.Scale.Max = cycles;
            pane.XAxis.Scale.Min = 0;
            pane.YAxis.Scale.Min = 0;
            pane.CurveList.Clear();
            pane.Legend.IsVisible = false;
            pane.AxisChange();
            errorGraph.Invalidate();
        }

        private void Test(object sender, EventArgs e)
        {
            if (xorNetwork != null)
            {
                lblTestOutput.Text = xorNetwork.Run(
                new double[] {double.Parse(txtTestInput.Text.Substring(2,4)),
                    double.Parse(txtTestInput.Text.Substring(8,4))})[0].ToString("0.000000");
            }
        }
    }
}

Its not to do with normalisation, as the mapping between the two sets of outputs is not monotonic. For example, the output in {0,1} is higher in EndSampleEvent but in {1,1} it is lower. Normalisation would be a simple linear function.

Its not to do with jitter either, as I've tried turning that off, and the results are still different.

Answer 1

I have received an answer from my professor. The problem lies in the LearnSample method from the BackpropagationNetwork class which is called for each training sample every iteration.

The order of relevant events in this method is …. 1) Add to the MeanSquaredError which is calculated using only the output layer and desired output 2) Backpropagate errors to all earlier layer; this has no effect on the network. 3) Finally recalculate biases for each layer; this affects the network.

(3) is the last thing that occurs in the LearnSample method and happens after the calculation of the output error for each training instance. For the XOR example this means that the network is changed 4 times from the state is was in when the MSE calculation was made.

In theory, if you want to compare training and test errors then you should do a manual calculation (like my GetError function) and run it twice: once for each data set. However, in reality it might not be necessary to go to all this trouble as the values are not that different.