no kernel image is available for execution on the device

Question

I training maskrcnn ,use tf-1.2 can train, but I use tf-1.5 it not training

The error is as follows：

Caused by op u'pyramid_1/AssignGTBoxes/Where_6', defined at:
  File "/home/zhouzd2/letrain/applications/letrain.py", line 349, in <module>
    tf.app.run()
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 124, in run
    _sys.exit(main(argv))
  File "/home/zhouzd2/letrain/applications/letrain.py", line 346, in main
    LeTrain().model_train(user_mode)
  File "/home/zhouzd2/letrain/platform/base_train.py", line 1228, in model_train
    cluster=self.cluster_spec)
  File "/home/zhouzd2/letrain/platform/deployment/model_deploy.py", line 226, in create_clones
    outputs, feed_ops,verify_model_loss = model_fn(*args, **kwargs)
  File "/home/zhouzd2/letrain/platform/base_train.py", line 1195, in clone_fn
    model_loss, end_points, feed_ops = network_fn(data_direct, data_batch, int_network_fn)
  File "/home/zhouzd2/letrain/applications/letrain.py", line 214, in get_loss
    FLAGS.batch_size)
  File "/home/zhouzd2/letrain/applications/fmrcnn/get_fmrcnn_loss.py", line 23, in model_fn
    loss_weights=[0.2, 0.2, 1.0, 0.2, 1.0])
  File "/home/zhouzd2/letrain/applications/fmrcnn/libs/nets/pyramid_network.py", line 580, in build
    is_training=is_training, gt_boxes=gt_boxes)
  File "/home/zhouzd2/letrain/applications/fmrcnn/libs/nets/pyramid_network.py", line 263, in build_heads
    assign_boxes(rois, [rois, batch_inds], [2, 3, 4, 5])
  File "/home/zhouzd2/letrain/applications/fmrcnn/libs/layers/wrapper.py", line 173, in assign_boxes
    inds = tf.where(tf.equal(assigned_layers, l))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/array_ops.py", line 2538, in where
    return gen_array_ops.where(condition=condition, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_ops.py", line 6087, in where
    "Where", input=condition, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 3160, in create_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1625, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-access

InternalError (see above for traceback): WhereOp: Could not launch cub::DeviceReduce::Sum to count number of true / nonzero indices.  temp_storage_bytes: 1, status: no kernel image is available for execution on the device
     [[Node: pyramid_1/AssignGTBoxes/Where_6 = Where[T=DT_BOOL, _device="/job:worker/replica:0/task:0/device:GPU:0"](pyramid_1/AssignGTBoxes/Equal_6_S9493)]]
     [[Node: pyramid_1/AssignGTBoxes/Reshape_8_G1028 = _Recv[client_terminated=false, recv_device="/job:worker/replica:0/task:0/device:CPU:0", send_device="/job:worker/replica:0/task:0/device:GPU:0", send_device_incarnation=5407481677180697062, tensor_name="edge_1349_pyramid_1/AssignGTBoxes/Reshape_8", tensor_type=DT_INT64, _device="/job:worker/replica:0/task:0/device:CPU:0"]()]]

No problem when loading calculation graphs, error is reported in sess.run()。 Does anyone know how to solve this problem? Or does anyone know what function can replace tf.where? Thank you！

Answer 1

If you are using Visual Studio:

Right click on the project > Properies > Cuda C/C++ > Device

and add the following to Code Generation field

compute_30,sm_30;compute_35,sm_35;compute_37,sm_37;compute_50,sm_50;compute_52,sm_52;compute_60,sm_60;compute_61,sm_61;compute_70,sm_70;compute_75,sm_75;