How to determine maximum batch size for a seq2seq tensorflow RNN training model

Question

Currently, I am using the default 64 as the batch size for the seq2seq tensorflow model. What is the maximum batch size , layer size etc I can go with a single Titan X GPU with 12 GB RAM with Haswell-E xeon 128GB RAM. The input data is converted to embeddings. Following are some helpful parameters I am using , it seems the cell input size is 1024:

encoder_inputs: a list of 2D Tensors [batch_size x cell.input_size].
 decoder_inputs: a list of 2D Tensors [batch_size x cell.input_size].
 tf.app.flags.DEFINE_integer("size", 1024, "Size of each model layer.")

So based on my hardware what is the maximum batch size , layers, input size I can go? Currently the GPU shows that 99% memory is occupied.

Answer 1

By default, Tensorflow occupies all GPU memory available. However, there is a way to change this. In my model, I do this:

config = tf.ConfigProto()
config.gpu_options.allow_growth = True

Then you can use this config when you start your session:

with tf.Session(config=config) as sess:

Now, the model will only use as much memory as it needs, and then you can try with different batch sizes and see when it runs out of memory.