Object Detection crash after 5428 steps, TypeError: 'numpy.float64' object cannot be interpreted as an integer

Question

My object detector has run multiple times but at this mark of 5428 it then crashes from TypeError's

I'm running in in anaconda with:

• numpy 1.18.1
• numpy-base 1.18.1
• tensorflow-gpu 1.14

This snippet below I think is the most important error?:

2020-02-19 13:56:06.901096: W tensorflow/core/framework/op_kernel.cc:1490] Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 117, in linspace
    num = operator.index(num)

TypeError: 'numpy.float64' object cannot be interpreted as an integer

The full traceback is below:

I0219 13:55:41.016854 15428 basic_session_run_hooks.py:260] loss = 0.0140173, step = 5400 (10.773 sec)
INFO:tensorflow:Saving checkpoints for 5428 into training/model.ckpt.
I0219 13:55:43.900022 15428 basic_session_run_hooks.py:606] Saving checkpoints for 5428 into training/model.ckpt.
INFO:tensorflow:Calling model_fn.
I0219 13:55:56.207441 15428 estimator.py:1145] Calling model_fn.
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:58.009801 15428 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:58.025418 15428 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:depth of additional conv before box predictor: 0
I0219 13:55:58.025418 15428 convolutional_box_predictor.py:151] depth of additional conv before box predictor: 0
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:59.573186 15428 regularizers.py:98] Scale of 0 disables regularizer.
INFO:tensorflow:Scale of 0 disables regularizer.
I0219 13:55:59.588815 15428 regularizers.py:98] Scale of 0 disables regularizer.
WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\eval_util.py:796: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
W0219 13:56:00.855241 15428 deprecation.py:323] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\eval_util.py:796: to_int64 (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.
Instructions for updating:
Use `tf.cast` instead.
WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\utils\visualization_utils.py:498: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

W0219 13:56:01.105266 15428 deprecation.py:323] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\utils\visualization_utils.py:498: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.
Instructions for updating:
tf.py_func is deprecated in TF V2. Instead, there are two
    options available in V2.
    - tf.py_function takes a python function which manipulates tf eager
    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to
    an ndarray (just call tensor.numpy()) but having access to eager tensors
    means `tf.py_function`s can use accelerators such as GPUs as well as
    being differentiable using a gradient tape.
    - tf.numpy_function maintains the semantics of the deprecated tf.py_func
    (it is not differentiable, and manipulates numpy arrays). It drops the
    stateful argument making all functions stateful.

WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\utils\visualization_utils.py:1044: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.

W0219 13:56:01.277014 15428 deprecation_wrapper.py:119] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\utils\visualization_utils.py:1044: The name tf.summary.image is deprecated. Please use tf.compat.v1.summary.image instead.

WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\model_lib.py:484: The name tf.metrics.mean is deprecated. Please use tf.compat.v1.metrics.mean instead.

W0219 13:56:01.386395 15428 deprecation_wrapper.py:119] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\model_lib.py:484: The name tf.metrics.mean is deprecated. Please use tf.compat.v1.metrics.mean instead.

INFO:tensorflow:Done calling model_fn.
I0219 13:56:01.749697 15428 estimator.py:1147] Done calling model_fn.
INFO:tensorflow:Starting evaluation at 2020-02-19T13:56:01Z
I0219 13:56:01.781106 15428 evaluation.py:255] Starting evaluation at 2020-02-19T13:56:01Z
INFO:tensorflow:Graph was finalized.
I0219 13:56:02.489665 15428 monitored_session.py:240] Graph was finalized.
2020-02-19 13:56:02.508162: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1640] Found device 0 with properties:
name: GeForce GTX 1080 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.683
pciBusID: 0000:06:00.0
2020-02-19 13:56:02.512995: I tensorflow/stream_executor/platform/default/dlopen_checker_stub.cc:25] GPU libraries are statically linked, skip dlopen check.
2020-02-19 13:56:02.516493: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1763] Adding visible gpu devices: 0
2020-02-19 13:56:02.518703: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1181] Device interconnect StreamExecutor with strength 1 edge matrix:
2020-02-19 13:56:02.523922: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1187]      0
2020-02-19 13:56:02.526614: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1200] 0:   N
2020-02-19 13:56:02.529223: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1326] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 8788 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1080 Ti, pci bus id: 0000:06:00.0, compute capability: 6.1)
WARNING:tensorflow:From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
W0219 13:56:02.535778 15428 deprecation.py:323] From C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
INFO:tensorflow:Restoring parameters from training/model.ckpt-5428
I0219 13:56:02.538779 15428 saver.py:1280] Restoring parameters from training/model.ckpt-5428
INFO:tensorflow:Running local_init_op.
I0219 13:56:03.495252 15428 session_manager.py:500] Running local_init_op.
INFO:tensorflow:Done running local_init_op.
I0219 13:56:03.656017 15428 session_manager.py:502] Done running local_init_op.
INFO:tensorflow:Performing evaluation on 5 images.
I0219 13:56:06.852077 13368 coco_evaluation.py:205] Performing evaluation on 5 images.
creating index...
index created!
INFO:tensorflow:Loading and preparing annotation results...
I0219 13:56:06.867704 13368 coco_tools.py:115] Loading and preparing annotation results...
INFO:tensorflow:DONE (t=0.00s)
I0219 13:56:06.867704 13368 coco_tools.py:137] DONE (t=0.00s)
creating index...
index created!
2020-02-19 13:56:06.901096: W tensorflow/core/framework/op_kernel.cc:1490] Invalid argument: TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.
Traceback (most recent call last):

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 117, in linspace
    num = operator.index(num)

TypeError: 'numpy.float64' object cannot be interpreted as an integer


During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 209, in __call__
    ret = func(*args)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py", line 384, in first_value_func
    self._metrics = self.evaluate()

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py", line 215, in evaluate
    coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_tools.py", line 176, in __init__
    iouType=iou_type)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 76, in __init__
    self.params = Params(iouType=iouType) # parameters

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 527, in __init__
    self.setDetParams()

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 507, in setDetParams
    self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)

  File "<__array_function__ internals>", line 6, in linspace

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 121, in linspace
    .format(type(num)))

TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.


Traceback (most recent call last):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1356, in _do_call
    return fn(*args)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1341, in _run_fn
    options, feed_dict, fetch_list, target_list, run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1429, in _call_tf_sessionrun
    run_metadata)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
         [[{{node IteratorGetNext}}]]
  (1) Out of range: End of sequence
         [[{{node IteratorGetNext}}]]
         [[Loss/BoxClassifierLoss/assert_equal/Assert/Assert/data_4/_2449]]
0 successful operations.
0 derived errors ignored.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\evaluation.py", line 272, in _evaluate_once
    session.run(eval_ops, feed_dict)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1252, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1353, in run
    raise six.reraise(*original_exc_info)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\six.py", line 703, in reraise
    raise value
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1338, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1411, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1169, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 950, in run
    run_metadata_ptr)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1173, in _run
    feed_dict_tensor, options, run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1350, in _do_run
    run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\client\session.py", line 1370, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.OutOfRangeError: 2 root error(s) found.
  (0) Out of range: End of sequence
         [[node IteratorGetNext (defined at model_main.py:105) ]]
  (1) Out of range: End of sequence
         [[node IteratorGetNext (defined at model_main.py:105) ]]
         [[Loss/BoxClassifierLoss/assert_equal/Assert/Assert/data_4/_2449]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'IteratorGetNext':
  File "model_main.py", line 109, in <module>
    tf.app.run()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
    return executor.run()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
    return self.run_local()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 367, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1158, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1192, in _train_model_default
    saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1484, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1252, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1338, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1419, in run
    run_metadata=run_metadata))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 594, in after_run
    if self._save(run_context.session, global_step):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 619, in _save
    if l.after_save(session, step):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 519, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 539, in _evaluate
    self._evaluator.evaluate_and_export())
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 920, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 477, in evaluate
    name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 519, in _actual_eval
    return _evaluate()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 501, in _evaluate
    self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1501, in _evaluate_build_graph
    self._call_model_fn_eval(input_fn, self.config))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1534, in _call_model_fn_eval
    input_fn, ModeKeys.EVAL)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1022, in _get_features_and_labels_from_input_fn
    self._call_input_fn(input_fn, mode))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\util.py", line 65, in parse_input_fn_result
    result = iterator.get_next()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\data\ops\iterator_ops.py", line 426, in get_next
    output_shapes=self._structure._flat_shapes, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\gen_dataset_ops.py", line 1947, in iterator_get_next
    output_shapes=output_shapes, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op
    op_def=op_def)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()



During handling of the above exception, another exception occurred:


Traceback (most recent call last):

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 209, in __call__
    ret = func(*args)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py", line 384, in first_value_func
    self._metrics = self.evaluate()

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py", line 215, in evaluate
    coco_wrapped_groundtruth, coco_wrapped_detections, agnostic_mode=False)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_tools.py", line 176, in __init__
    iouType=iou_type)

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 76, in __init__
    self.params = Params(iouType=iouType) # parameters

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 527, in __init__
    self.setDetParams()

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\pycocotools\cocoeval.py", line 507, in setDetParams
    self.iouThrs = np.linspace(.5, 0.95, np.round((0.95 - .5) / .05) + 1, endpoint=True)

  File "<__array_function__ internals>", line 6, in linspace

  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\numpy\core\function_base.py", line 121, in linspace
    .format(type(num)))

TypeError: object of type <class 'numpy.float64'> cannot be safely interpreted as an integer.


         [[node PyFunc_3 (defined at C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py:394) ]]
0 successful operations.
0 derived errors ignored.

Original stack trace for 'PyFunc_3':
  File "model_main.py", line 109, in <module>
    tf.app.run()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\platform\app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 299, in run
    _run_main(main, args)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\absl\app.py", line 250, in _run_main
    sys.exit(main(argv))
  File "model_main.py", line 105, in main
    tf.estimator.train_and_evaluate(estimator, train_spec, eval_specs[0])
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 473, in train_and_evaluate
    return executor.run()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 613, in run
    return self.run_local()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 714, in run_local
    saving_listeners=saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 367, in train
    loss = self._train_model(input_fn, hooks, saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1158, in _train_model
    return self._train_model_default(input_fn, hooks, saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1192, in _train_model_default
    saving_listeners)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1484, in _train_with_estimator_spec
    _, loss = mon_sess.run([estimator_spec.train_op, estimator_spec.loss])
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 754, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1252, in run
    run_metadata=run_metadata)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1338, in run
    return self._sess.run(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\monitored_session.py", line 1419, in run
    run_metadata=run_metadata))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 594, in after_run
    if self._save(run_context.session, global_step):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\training\basic_session_run_hooks.py", line 619, in _save
    if l.after_save(session, step):
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 519, in after_save
    self._evaluate(global_step_value)  # updates self.eval_result
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 539, in _evaluate
    self._evaluator.evaluate_and_export())
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\training.py", line 920, in evaluate_and_export
    hooks=self._eval_spec.hooks)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 477, in evaluate
    name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 519, in _actual_eval
    return _evaluate()
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 501, in _evaluate
    self._evaluate_build_graph(input_fn, hooks, checkpoint_path))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1501, in _evaluate_build_graph
    self._call_model_fn_eval(input_fn, self.config))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1537, in _call_model_fn_eval
    features, labels, ModeKeys.EVAL, config)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow_estimator\python\estimator\estimator.py", line 1146, in _call_model_fn
    model_fn_results = self._model_fn(features=features, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\model_lib.py", line 482, in model_fn
    eval_config, list(category_index.values()), eval_dict)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\eval_util.py", line 947, in get_eval_metric_ops_for_evaluators
    eval_dict))
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\object_detection-0.1-py3.7.egg\object_detection\metrics\coco_evaluation.py", line 394, in get_estimator_eval_metric_ops
    first_value_op = tf.py_func(first_value_func, [], tf.float32)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 324, in new_func
    return func(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 480, in py_func
    return py_func_common(func, inp, Tout, stateful, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 462, in py_func_common
    func=func, inp=inp, Tout=Tout, stateful=stateful, eager=False, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\script_ops.py", line 285, in _internal_py_func
    input=inp, token=token, Tout=Tout, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\ops\gen_script_ops.py", line 159, in py_func
    "PyFunc", input=input, token=token, Tout=Tout, name=name)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\op_def_library.py", line 788, in _apply_op_helper
    op_def=op_def)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\util\deprecation.py", line 507, in new_func
    return func(*args, **kwargs)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 3616, in create_op
    op_def=op_def)
  File "C:\Users\luke9\Anaconda3\envs\tf_gpu\lib\site-packages\tensorflow\python\framework\ops.py", line 2005, in __init__
    self._traceback = tf_stack.extract_stack()

This is missing some of the traceback because of the character limit, but it is all numpy TypeError's.

Answer 1

Try downgrading your numpy version. In my case, i had to downgrade it to 1.17.4

Answer 2

I had the same issue. It seems that updating to tensorflow 1.15.0 solves it.