Encog: BasicNetwork: Online learning without preconstructed dataset

Question

I am trying to use the encog library as a function approximator for a reinforcement learning problem. To be more precise, I am trying to get a multi layer perceptron (BasicNetwork) up and running. Since my agent will somehow explore the world based on whatever RL-algorithm I chose I cannot prebuild any BasicNeuralDataSet as shown in the XOR example. Probably, I have to use the pause() and resume() functions but since I cannot find any documentation or examples on these I am somewhat lost in how to use these features (if they even work in my version. I'm not quite sure after reading the answer to the question in the second link).

I am using Java and the encog-core-2.5.3 jar. My current approach looks like this:

BasicNetwork network = new BasicNetwork();
network.addLayer(new BasicLayer(null, true,2));
network.addLayer(new BasicLayer(new ActivationTANH(), true,4));
network.addLayer(new BasicLayer(new ActivationTANH(), true,1));
network.getStructure().finalizeStructure();
network.reset();

TrainingContinuation cont = null;
double error = 0;

do {
    int rnd = random.nextInt(trainInputs.length);
    NeuralDataSet trainingSet = new BasicNeuralDataSet(
        new double[][] { trainInputs[rnd] },
        new double[][] { trainOutputs[rnd] });

    Backpropagation train = new Backpropagation(network, trainingSet);

    // train the neural network
    if (cont != null) {
        train.resume(cont);
    }

    train.iteration();
    cont = train.pause();

    error = train.getError();
} while (error > 0.01);

This is obviously a minimal example where I am just drawing random datapoints from a toy sample (XOR). What happens is that the MLP does not converge. Logging is showing me completely random errors so I assume that somewhat the trainer is being reset and that my pause/resume approach is not correctly implemented.

---

P.S.: Since I am not bound to Encoq but can use any framework there is I also appreciate sample code that fulfills my requirements. So far I tried Weka and Neuroph but both seem to lack real online learning where one can just trigger the training whenever a new sample is available (It has to be possible to classify samples during any time as well)

Answer 1

Sorry about the slow response. Basically, it sounds like you are asking for online training. That is you just present a single case and the neural network weights are immediately updated. That way it would not be necessary to create an entire training set, you just train as needed. Unfortunately, Encog does not have good support of this. It has become a frequently asked question, and I plan to add it in the next release.

For now, about the only way you could do it is to create a training set with a single item and then train for a single iteration.

EDIT Online training has been added as of Encog 3.2. See this FAQ for more info.

http://www.heatonresearch.com/faq/5/3