How to encode input for a tensorflow model having encoded_image_string_tensor as input

Question

I trained an object detection model on Google AI platform and downloaded the model. It's a standard saved_model.pb file which I want to load in python and provide one image for inference. The problem is that the input of this model is defined as encoded_image_string_tensor which expects a base64 encoded string. How can I encode an image file in this format in python?

print(model.inputs)
print(model.output_dtypes)
print(model.output_shapes)

[<tf.Tensor 'encoded_image_string_tensor:0' shape=(None,) dtype=string>, <tf.Tensor 'key:0' shape=(None,) dtype=string>, <tf.Tensor 'global_step:0' shape=() dtype=resource>]
{'detection_scores': tf.float32, 'detection_classes': tf.float32, 'num_detections': tf.float32, 'key': tf.string, 'detection_boxes': tf.float32}
{'detection_scores': TensorShape([None, 100]), 'detection_classes': TensorShape([None, 100]), 'num_detections': TensorShape([None]), 'key': TensorShape([None]), 'detection_boxes': TensorShape([None, 100, 4])}

The existing examples in tensorflow/models/research show how to do it with an image_tensor type of input:

  image = np.asarray(image)
  # The input needs to be a tensor, convert it using `tf.convert_to_tensor`.
  input_tensor = tf.convert_to_tensor(image)
  # The model expects a batch of images, so add an axis with `tf.newaxis`.
  input_tensor = input_tensor[tf.newaxis,...]

When I run this code on the model with encoded_image_string_tensor as an input it produces the following error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
 in 
      1 for i in range(1):
----> 2     show_inference(model, TEST_IMAGE_PATHS[i])

 in show_inference(model, image_path)
     39 #   print(image_np)
     40   # Actual detection.
---> 41   output_dict = run_inference_for_single_image(model, image_np)
     42   # Visualization of the results of a detection.
     43   print(output_dict['detection_scores'][:3])

 in run_inference_for_single_image(model, image)
      7 
      8   # Run inference
----> 9   output_dict = model(input_tensor)
     10 
     11   # All outputs are batches tensors.

~\anaconda3\envs\tf2\lib\site-packages\tensorflow\python\eager\function.py in __call__(self, *args, **kwargs)
   1603       TypeError: For invalid positional/keyword argument combinations.
   1604     """
-> 1605     return self._call_impl(args, kwargs)
   1606 
   1607   def _call_impl(self, args, kwargs, cancellation_manager=None):

~\anaconda3\envs\tf2\lib\site-packages\tensorflow\python\eager\function.py in _call_impl(self, args, kwargs, cancellation_manager)
   1622            "of {}), got {}. When calling a concrete function, positional "
   1623            "arguments may not be bound to Tensors within nested structures."
-> 1624           ).format(self._num_positional_args, self._arg_keywords, args))
   1625     args = list(args)
   1626     for keyword in self._arg_keywords[len(args):]:

TypeError: Expected at most 0 positional arguments (and the rest keywords, of ['encoded_image', 'key']), got (,). When calling a concrete function, positional arguments may not be bound to Tensors within nested structures.

Answer 1

You can easily adapt the run_inference_for_single_image() function from the object_detection_tutorial.ipynb notebook (which you seem to use in your example), by using tf.io.encode_jpeg() to encode the image.

The built-in object detection model from Google AI platform also requires a key (any string-tensor with dimension of batch-size) as input, which I've also added to the model() call in the example below .

def run_inference_for_single_image(model, image):
    image = np.asarray(image)

    # The input needs to be a tensor, convert it using `tf.convert_to_tensor`.
    input_tensor = tf.convert_to_tensor(image)

    # Encode the (numerical) tensor into an "encoded_image"
    encoded_image = tf.io.encode_jpeg(input_tensor)

    # The model expects a batch of images, so add an axis with `tf.newaxis`.
    encoded_image = encoded_image[tf.newaxis,...]

    # Run inference (the SavedModel downloaded from AI platform also requires a "key" as input.)
    output_dict = model(encoded_image = encoded_image, key = tf.expand_dims(tf.convert_to_tensor("test_key"), 0))

    # ...