Repeating training with the same data set

Question

I have recently been learning about TensorFlow and I'm trying to put together a network with the following format:

n_inputs = 16
n_hidden = 64*2
n_outputs = 3

x = tf.placeholder(tf.float32, shape=[1,n_inputs])
W = tf.Variable(tf.truncated_normal([n_inputs, n_hidden]))
b = tf.Variable(tf.zeros([n_hidden]))
hidden_layer = tf.nn.relu(tf.matmul(x, W) + b)

W2 = tf.Variable(tf.truncated_normal([n_hidden, n_outputs]))
b2 = tf.Variable(tf.zeros([n_outputs]))

logits = tf.matmul(hidden_layer, W2) + b2

I only have 1784 sets of training data, is it valid to repeat my training using this data repeatedly? I would guess that it will result in over-fitting to the training data if repeated too many times.

I am currently training like this:

print "Training"
for i in range(100):
    errs = []
    for xt, yt in zip(train[:n_dat-50], test[:n_dat-50]):
        _, err = sess.run([train_step, cost], feed_dict={x: [xt], y_: [yt]})
        errs.append(err)
    print "Error: %.5f" % np.mean(errs)

I'm also looking into using L2 regularisation and Dropouts to help improve the classification. Any other tips to improve training with low levels of data would be very helpful.

Answer 1

You might consider adding noise. Just add random stuff to your inputs (maybe have the noise have the same mean and variance - depends). This prevents some overfittig and gives you "more" training data (make sure you put some data to the side in order for you to validate the generalized training success).

Also, sometimes, it is possible to create artificial data-sets that follow the same logic for some pretraining.