What is the way to use Tensor flow 2.0 object in open cv2 python and why is it so circuitous?

Question

I load an image using tensor flow api (2.0) like so :

def load(image_file):
  image = tf.io.read_file(image_file)
  image = tf.image.decode_jpeg(image)

Now that I have this object, I want to show this image, I can simply use matplotlib.pyplot, and this works.

plt.figure()
plt.imshow(re/255.0)
plt.show()

However attempting this with OpenCV2 is problematic from the start, most of the examples are from 1.0 with .eval() session based suggestion for numpy conversion. One way would be to first convert tensor flow object to numpy, here is the function to do that from API documentation :

TensorFlow
API r2.0
TensorFlow Core 2.0a
Python
tf.make_ndarray
Create a numpy ndarray from a tensor.

I dont understand why this does not works and I get a number of errors while all I want is to do something simple and then use some open cv2 functions like remap, resize etc.:

File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\def_function.py", line 426, in call self._initialize(args, kwds, add_initializers_to=initializer_map) File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\def_function.py", line 370, in _initialize *args, **kwds)) File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\function.py", line 1313, in _get_concrete_function_internal_garbage_collected graph_function, _, _ = self._maybe_define_function(args, kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\function.py", line 1580, in _maybe_define_function graph_function = self._create_graph_function(args, kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\function.py", line 1512, in _create_graph_function capture_by_value=self._capture_by_value), File "C:\Python\Python37\lib\site-packages\tensorflow\python\framework\func_graph.py", line 694, in func_graph_from_py_func func_outputs = python_func(*func_args, **func_kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\eager\def_function.py", line 317, in wrapped_fn return weak_wrapped_fn().wrapped(*args, **kwds) File "C:\Python\Python37\lib\site-packages\tensorflow\python\framework\func_graph.py", line 686, in wrapper ), args, kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 392, in converted_call result = converted_f(*effective_args, **kwargs) File "C:\Users\syeda\AppData\Local\Temp\tmpnahp3og4.py", line 32, in tf__random_deform im2 = ag__.converted_call('make_ndarray', tf, ag__.ConversionOptions(recursive=True, verbose=0, strip_decorators=(tf.function, defun_9, ag__.convert, ag__.do_not_convert, ag__.converted_call), force_conversion=False, optional_features=(), internal_convert_user_code=True), (real_image,), {}) File "C:\Python\Python37\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 267, in converted_call return _call_unconverted(f, args, kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\autograph\impl\api.py", line 188, in _call_unconverted return f(*args, **kwargs) File "C:\Python\Python37\lib\site-packages\tensorflow\python\framework\tensor_util.py", line 596, in MakeNdarray shape = [d.size for d in tensor.tensor_shape.dim] AttributeError: 'Tensor' object has no attribute 'tensor_shape'

Update 5/5/2018 : After searching more I found out that this has something to do with Tensorflow graph execution. I have a function

def load_image_train(image_file):
  input_image, real_image = load(image_file)
 print(type(real_image))
  print(real_image.shape)
  some_image = Open CV operations like filtering, jitter etc performed on real_image
return some_image

This works nicely when called eagerly with .numpy() attribute, however when called like following code and when you try to inspect what real_image is and its type returns

class 'tensorflow.python.framework.ops.Tensor' (None, None, None)

Please advice.

# Input pipeline
train_dataset = tf.data.Dataset.list_files(PATH+'train/*.jpg')
train_dataset = train_dataset.shuffle(BUFFER_SIZE)
train_dataset = train_dataset.map(load_image_train,
                               num_parallel_calls=tf.data.experimental.AUTOTUNE)
train_dataset = train_dataset.batch(1)

Update 5/5/2018 : I decided to do a preprocessing of the data so I don't have to worry about the using any opencv functionality during the load time of the data. However during training time I still want to do some openCV operations. Now as per the suggestion of @giser_yugang I tried using py_function, I wrap opencv operations in py_function and call that function in a wrapper tf.function. This wrapper tf.function I call in train step. However the output I get from this wrapper function is like so :

class 'tensorflow.python.framework.ops.Tensor'
unknown

Then if I try to consume this tensor in the next train step operation I get a

incompatible with the layer: its rank is undefined, but the layer requires a defined rank.

If I don't use this py_function wrapper in my train step and directly try the numpy operations using opencv I get another error

AttributeError: 'Tensor' object has no attribute 'numpy'

I guess both ways you cant win !

Answer 1

Using OpenCV + Tensorflow 2.0 is straightforward.

Let's say we have the image "wink.jpg" (see wink image attached) in the current directory, then it is possible to use Tensorflow 2.0 to read the JPEG image and get a tf.Tensor with dtype=uint8, get a numpy array from it and use OpenCV to visualize it (in BGR format, as OpenCV needs).

import tensorflow as tf
import cv2


def load(image_file):
    image = tf.io.read_file(image_file)
    image = tf.image.decode_jpeg(image)
    return image


wink = load("wink.jpg")
print(wink.shape, wink.dtype)

# Get a Numpy BGR image from a RGB tf.Tensor
image = cv2.cvtColor(wink.numpy(), cv2.COLOR_RGB2BGR)

cv2.imshow("image", image)
cv2.waitKey()

If you have some problem related to the Graph architecture, it is probably because you're:

• Or using tf.function to convert the code the graph (in this case just remove the annotation)
• Or using OpenCV inside a tf.data.Dataset method (in this case just do not use OpenCV or use tf.py_func where OpenCV is needed)
• Or using a wrong version of Tensorflow, that is not 2 with eager mode enabled by default (check if everything is ok by running pip list |grep tf and pip list | grep tensor and if you see something weird like more than 1 version of TF installed, I suggest to delete the environment a start with a new installation).