Some Python objects were not bound to checkpointed values

Question

I am trying to get started with Tensorflow 2.0 Object Detection API. I have gone through the installation following the official tutorial and I pass all the tests. However, I keep getting an error message that I don't understand when I try to run the main module. This is how I run it:

python model_main_tf2.py --model_dir=ssd_resnet50_v1_fpn_640x640_coco17_tpu-8 --pipeline_config_path=ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/pipeline.config

This is the beginning of the error message:

Traceback (most recent call last):
  File "model_main_tf2.py", line 113, in <module>
    tf.compat.v1.app.run()
  File "/home/hd/hd_hd/hd_rs239/.conda/envs/jan_tf2/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/home/hd/hd_hd/hd_rs239/.conda/envs/jan_tf2/lib/python3.7/site-packages/absl/app.py", line 299, in run
    _run_main(main, args)
  File "/home/hd/hd_hd/hd_rs239/.conda/envs/jan_tf2/lib/python3.7/site-packages/absl/app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "model_main_tf2.py", line 110, in main
    record_summaries=FLAGS.record_summaries)
  File "/home/hd/hd_hd/hd_rs239/.conda/envs/jan_tf2/lib/python3.7/site-packages/object_detection/model_lib_v2.py", line 569, in train_loop
    unpad_groundtruth_tensors)
  File "/home/hd/hd_hd/hd_rs239/.conda/envs/jan_tf2/lib/python3.7/site-packages/object_detection/model_lib_v2.py", line 383, in load_fine_tune_checkpoint
    ckpt.restore(checkpoint_path).assert_existing_objects_matched()
  File "/home/hd/hd_hd/hd_rs239/.conda/envs/jan_tf2/lib/python3.7/site-packages/tensorflow/python/training/tracking/util.py", line 791, in assert_existing_objects_matched
    (list(unused_python_objects),))
AssertionError: Some Python objects were not bound to checkpointed values, likely due to changes in the Python program: [SyncOnReadVariable:{
  0: <tf.Variable 'conv2_block1_0_bn/moving_variance:0' shape=(256,) dtype=float32, numpy=
array([1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1., 1.,

In the pipeline.config, I specify a checkpoint like this:

  fine_tune_checkpoint: "ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/checkpoint/ckpt-0" 

These are the contents of ssd_resnet50_v1_fpn_640x640_coco17_tpu-8/checkpoint/ :

checkpoint  
ckpt-0.data-00000-of-00001  
ckpt-0.index

I have searched Google but couldn't find any answer. In this issue, the suggested solution is outdated (the code they suggest to replace is not there anymore).

Question: What is the problem and how can I solve it?

I am doing this on a server with CentOS Linux 7. I am using Python 3.7. I am new to Tensorflow so please if I am missing any important information, let me know.

Answer 1

From the file name you provided (ssd_resnet50_v1_fpn_640x640_coco17_tpu-8), I can see you are trying to work with an object detection task. Therefore, in your pipeline.config file change this line:

fine_tune_checkpoint_type: "classification"

To:

fine_tune_checkpoint_type: "detection"

This should solve your problem.

Answer 2

For me it was usefull to check type of feature extractor. I change type: "mobilenet_v2" to type: "mobilenet_v2_fpn_sep_conv" in pipeline.config. And its start working.