Online or batch training by default in tensorflow

Question

I have the following question: I'm trying to learn tensor-flow and I still don't find where to set the training as online or batch. For example, if I have the following code to train a neural-network:

loss_op = tf.reduce_mean(tf.pow(neural_net(X) - Y, 2))
optimizer = tf.train.GradientDescentOptimizer(learning_rate=learning_rate)
train_op = optimizer.minimize(loss_op)


sess.run(train_op, feed_dict={X: batch_x, Y: batch_y})

If I give all the data at the same time (i.e batch_x has all the data), does that mean that is training as a batch training? or the tensor-flow optimizer optimize in a different way from behind? Is it wrong if I do a for loop giving one data sample at a time? does that count as single-step (online) training? Thank you for your help.

Answer 1

There are mainly 3 Types of Gradient Descent. Specifically,

1. Stochastic Gradient Descent
2. Batch Gradient Descent
3. Mini Batch Gradient Descent

Here, is a good tutorial (https://machinelearningmastery.com/gentle-introduction-mini-batch-gradient-descent-configure-batch-size/) on above three methods with upsides and downsides.

For your question, Following is a standard sample training tensorflow code,

N_EPOCHS = #Need to define here
BATCH_SIZE = # Need to define hare

with tf.Session() as sess:
   train_count = len(train_x)    

    for i in range(1, N_EPOCHS + 1):
        for start, end in zip(range(0, train_count, BATCH_SIZE),
                              range(BATCH_SIZE, train_count + 1,BATCH_SIZE)):

            sess.run(train_op, feed_dict={X: train_x[start:end],
                                           Y: train_y[start:end]})

Here N_EPOCHS means the number of passes of the whole training dataset. And you can set the BATCH_SIZE according to your Gradient Descent method.

• For Stochastic Gradient Descent, BATCH_SIZE = 1.

• For Batch Gradient Descent, BATCH_SIZE = training dataset size.

• For Mini Batch Gradient Decent, 1 << BATCH_SIZE << training dataset size.

Among three methods, the most popular method is the Mini Batch Gradient Decent. However, you need to set the BATCH_SIZE parameter according to your requirements. A good default for BATCH_SIZE might be 32.

Hope this helps.