Tensorflow tf.train.Saver saves suspiciously large .ckpt files?

Question

I'm working with a reasonably sized net (1 convolutional layer, 2 fully connected layers). Every time I save variables using tf.train.Saver, the .ckpt files are half a gigabyte each of disk space (512 MB to be exact). Is this normal? I have a Caffe net with the same architecture that requires only a 7MB .caffemodel file. Is there a particular reason why Tensorflow saves such large file sizes?

Many thanks.

Answer 1

Hard to tell how large your net is from what you've described -- the number of connections between two fully connected layers scales up quadratically with the size of each layer, so perhaps your net is quite large depending on the size of your fully connected layers.

If you'd like to save space in the checkpoint files, you could replace this line:

saver = tf.train.Saver()

with the following:

saver = tf.train.Saver(tf.trainable_variables())

By default, tf.train.Saver() saves all variables in your graph -- including the variables created by your optimizer to accumulate gradient information. Telling it to save only trainable variables means it will save only the weights and biases of your network, and discard the accumulated optimizer state. Your checkpoints will probably be a lot smaller, with the tradeoff that it you may experience slower training for the first few training batches after you resume training, while the optimizer re-accumulates gradient information. It doesn't take long at all to get back up to speed, in my experience, so personally, I think the tradeoff is worth it for the smaller checkpoints.

Answer 2

Maybe you can try (in Tensorflow 1.0):

saver.save(sess, filename, write_meta_graph=False)

which doesn't save meta Graph information. See: https://www.tensorflow.org/versions/master/api_docs/python/tf/train/Saver https://www.tensorflow.org/programmers_guide/meta_graph