Tensorflow: ValueError: Can't load save_path when it is None in single shot detection

Question

I have just started working with tensorflow in python. I am trying to train Single shot detection using tensorflow for pascalvoc dataset. While creating tfrecords and during evaluation using VGG_VOC0712_SSD_300x300_ft_iter_120000.ckpt trained model there was no error. Whereas, when I am trying to train pascalvoc 2007 or 2012 datasets using ssd_300_vgg.ckpt pre-trained model I am getting following error.

2017-08-25 20:03:03.001268: I tensorflow/core/common_runtime gpu/gpu_device.cc:1045] Creating TensorFlow device (/gpu:0) -> (device: 0, name: Quadro M5000M, pci bus id: 0000:01:00.0)
INFO:tensorflow:Error reported to Coordinator: <type 'exceptions.ValueError'>, Can't load save_path when it is None.
Traceback (most recent call last):
  File "train_ssd_network.py", line 391, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 48, in run
    _sys.exit(main(_sys.argv[:1] + flags_passthrough))
  File "train_ssd_network.py", line 387, in main
    sync_optimizer=None)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/slim/python/slim/learning.py", line 738, in train
    master, start_standard_services=False, config=session_config) as sess:
  File "/usr/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 965, in managed_session
    self.stop(close_summary_writer=close_summary_writer)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 793, in stop
    stop_grace_period_secs=self._stop_grace_secs)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/coordinator.py", line 389, in join
    six.reraise(*self._exc_info_to_raise)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 954, in managed_session
    start_standard_services=start_standard_services)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/supervisor.py", line 709, in prepare_or_wait_for_session
    init_feed_dict=self._init_feed_dict, init_fn=self._init_fn)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/session_manager.py", line 281, in prepare_session
    init_fn(sess)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/contrib/framework/python/ops/variables.py", line 660, in callback
    saver.restore(session, model_path)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/training/saver.py", line 1558, in restore
    raise ValueError("Can't load save_path when it is None.")
ValueError: Can't load save_path when it is None.

I am using following script to fine-tune the model

DATASET_DIR=./tfrecords
TRAIN_DIR=./logs/
CHECKPOINT_PATH=./checkpoints/ssd_300_vgg.ckpt
python train_ssd_network.py \
    --train_dir=${TRAIN_DIR} \
    --dataset_dir=${DATASET_DIR} \
    --dataset_name=pascalvoc_2012 \
    --dataset_split_name=train \
    --model_name=ssd_300_vgg \
    --checkpoint_path=${CHECKPOINT_PATH} \
    --save_summaries_secs=60 \
    --save_interval_secs=600 \
    --weight_decay=0.0005 \
    --optimizer=adam \
    --learning_rate=0.001 \
    --batch_size=10

The model ssd_300_vgg.ckpt is stored at the location ./checkpoints

Please let me know if anyone has the solution.

Answer 1

check your path, you might be pointing to a non-existing file.

Answer 2

Three suggestions:

• Check the path when restoring the model

  saver = tf.train.import_meta_graph(model_path)

• Check the path when restoring the checkpoint

  saver.restore(sess, tf.train.latest_checkpoint(cur_dir))

• Check the parameters when saving the model

  saver = tf.train.Saver(save_relative_paths=True)
  

Answer 3

CHECKPOINT_PATH=./checkpoints/ssd_300_vgg.ckpt/ssd_300_vgg.ckpt