Neurolab retrain the network

Question

I am using neurolab in python to create a neural-netowork. I create a newff network and am using the default train_bfgs training function. My problem is a lot of times, the training just ends way before either the epochs run out or even the error target is reached. I looked around and found a post on neurolabs github page, where they kinda explained why this was happening. My problem is, if I rerun the program a few times it just catches-on and the training starts and then the error also falls (probably some random starting weights are a lot better then the others). What I want to do is to put a kind of check in the training so that if the error is too high and the epochs it trained are not even close to the total then retrain the network (sort of like rerunning the program) (maybe resetting the network default weights)

Here is what I have written, but obviously it doesnt work

trainingComplete = False
while not trainingComplete:
    error = net.train(trainingData, TS, epochs=50, show=10, goal=0.001)
    if len(error) < 0.8*epochs:
        if len(error) > 0 and min(error) < 0.01:
            trainingComplete = True
        else:
            net.reset()
            continue
    else:
        trainingComplete = True

what is going on is, when it passes the first condition, i.e. too few training epochs, it executes the net.reset() before restarting, but then on, there is no training that is happening and this becomes an infinite loop. Any idea what I am missing?

Thanks

Answer 1

So, Since this went un answered for a few days, and I think its really bad for SO so I took it upon my self to find a working work-around. I tired restarting the script by using the os.execv(__file__, sys.argv), but on my mac that is always a permission problem, plus its just too dirty, so here is how i get it to work now.

# Train network
print('Starting training....')
trainingComplete = False
while not trainingComplete:
    error = net.train(trainingData, TS, epochs=epochs, show=10, goal=0.001)
    if len(error) < 0.8 * epochs:
       if len(error) > 0 and min(error) < 0.01:
           trainingComplete = True
       else:
           print('Restarting....')
           net = createNeuralNetwork(trainingData, [hidden], 1)
           net.trainf = train_bfgs
    else:  
       trainingComplete = True

Its pretty hacky but kinda works:

Starting training....
Restarting....
Restarting....
Restarting....
Restarting....
Restarting....
Restarting....
Restarting....
Restarting....
Epoch: 10; Error: 1.46314116045;
Epoch: 20; Error: 0.759613243435;
Epoch: 30; Error: 0.529574731856;
.
.

Hope that helps some one