Convert python opencv mat image to tensorflow image data

Question

I want to capture frames from a video with python and opencv and then classify the captured Mat images with tensorflow. The problem is that i don´t know how to convert de Mat format to a 3D Tensor variable. This is how i am doing now with tensorflow (loading the image from file) :

image_data = tf.gfile.FastGFile(imagePath, 'rb').read()
with tf.Session() as sess:
    softmax_tensor = sess.graph.get_tensor_by_name('final_result:0')
    predictions = sess.run(softmax_tensor,
                           {'DecodeJpeg/contents:0': image_data})

I will appreciate any help, thanks in advance

Answer 1

With Tensorflow 2.0 and OpenCV 4.2.0, you can convert by this way :

import numpy as np
import tensorflow as tf
import cv2 as cv

width = 32
height = 32

#Load image by OpenCV
img = cv.imread('img.jpg')

#Resize to respect the input_shape
inp = cv.resize(img, (width , height ))

#Convert img to RGB
rgb = cv.cvtColor(inp, cv.COLOR_BGR2RGB)

#Is optional but i recommend (float convertion and convert img to tensor image)
rgb_tensor = tf.convert_to_tensor(rgb, dtype=tf.float32)

#Add dims to rgb_tensor
rgb_tensor = tf.expand_dims(rgb_tensor , 0)

#Now you can use rgb_tensor to predict label for exemple :

#Load pretrain model, made from: https://www.tensorflow.org/tutorials/images/cnn
model = tf.keras.models.load_model('cifar10_model.h5')

#Create probability model 
probability_model = tf.keras.Sequential([model, 
                                     tf.keras.layers.Softmax()])
#Predict label
predictions = probability_model.predict(rgb_tensor, steps=1)

Answer 2

Load the OpenCV image using imread, then convert it to a numpy array.

For feeding into inception v3, you need to use the Mult:0 Tensor as entry point, this expects a 4 dimensional Tensor that has the layout: [Batch index,Width,Height,Channel] The last three are perfectly fine from a cv::Mat, the first one just needs to be 0, as you do not want to feed a batch of images, but a single image. The code looks like:

#Loading the file
img2 = cv2.imread(file)
#Format for the Mul:0 Tensor
img2= cv2.resize(img2,dsize=(299,299), interpolation = cv2.INTER_CUBIC)
#Numpy array
np_image_data = np.asarray(img2)
#maybe insert float convertion here - see edit remark!
np_final = np.expand_dims(np_image_data,axis=0)

#now feeding it into the session:
#[... initialization of session and loading of graph etc]
predictions = sess.run(softmax_tensor,
                           {'Mul:0': np_final})
#fin! 

Kind regards,

Chris

Edit: I just noticed, that the inception network wants intensity values normalized as floats to [-0.5,0.5], so please use this code to convert them before building the RGB image:

np_image_data=cv2.normalize(np_image_data.astype('float'), None, -0.5, .5, cv2.NORM_MINMAX)