Inference time using of Tensorflow Object Detection

Question

I have deployed my object detection model to Google Kubernetes Engine. My model is trained using faster_rcnn_resnet101_pets configuration. The inference time of my model is very high (~10 seconds total time for prediction and ) even though I am using a Nvidia Tesla K80 GPU in my cluster node. I am using gRPC for getting predicitons from the model. The script for making prediciton requests is :

import argparse
import os
import time
import sys
import tensorflow as tf
from PIL import Image
import numpy as np
from grpc.beta import implementations
sys.path.append("..")
from object_detection.core.standard_fields import \
    DetectionResultFields as dt_fields
from object_detection.utils import label_map_util
from argparse import RawTextHelpFormatter
from tensorflow_serving.apis import predict_pb2
from tensorflow_serving.apis import prediction_service_pb2_grpc


tf.logging.set_verbosity(tf.logging.INFO)

WIDTH = 1024
HEIGHT = 768


def load_image_into_numpy_array(input_image):
    image = Image.open(input_image)
    image = image.resize((WIDTH, HEIGHT), Image.ANTIALIAS)
    (im_width, im_height) = image.size
    image_arr = np.array(image.getdata()).reshape(
        (im_height, im_width, 3)).astype(np.uint8)
    image.close()
    return image_arr


def load_input_tensor(input_image):

    image_np = load_image_into_numpy_array(input_image)
    image_np_expanded = np.expand_dims(image_np, axis=0).astype(np.uint8)
    tensor = tf.contrib.util.make_tensor_proto(image_np_expanded)
    return tensor


def main(args):
    start_main = time.time()

    host, port = args.server.split(':')

    channel = implementations.insecure_channel(host, int(port))._channel

    stub = prediction_service_pb2_grpc.PredictionServiceStub(channel)
    request = predict_pb2.PredictRequest()
    request.model_spec.name = args.model_name

    input_tensor = load_input_tensor(args.input_image)
    request.inputs['inputs'].CopyFrom(input_tensor)
    start = time.time()

    result = stub.Predict(request, 60.0)
    end = time.time()

    output_dict = {}

    output_dict[dt_fields.detection_classes] = np.squeeze(
        result.outputs[dt_fields.detection_classes].float_val).astype(np.uint8)
    output_dict[dt_fields.detection_boxes] = np.reshape(
        result.outputs[dt_fields.detection_boxes].float_val, (-1, 4))
    output_dict[dt_fields.detection_scores] = np.squeeze(
        result.outputs[dt_fields.detection_scores].float_val)
    category_index = label_map_util.create_category_index_from_labelmap(args.label_map,
                                                                        use_display_name=True)
    classes = output_dict[dt_fields.detection_classes]
    scores = output_dict[dt_fields.detection_scores]
    classes.shape = (1, 300)
    scores.shape = (1, 300)
    print("prediction time : " + str(end-start))
    objects = []

    threshold = 0.5  # in order to get higher percentages you need to lower this number; usually at 0.01 you get 100% predicted objects
    for index, value in enumerate(classes[0]):
        object_dict = {}
        if scores[0, index] > threshold:
            object_dict[(category_index.get(value)).get('name').encode('utf8')] = \
                scores[0, index]
            objects.append(object_dict)
    print(objects)
    end_main = time.time()

    print("Overall Time : " + str(end_main-start_main))


if __name__ == '__main__':
    parser = argparse.ArgumentParser(description="Object detection grpc client.",
                                     formatter_class=RawTextHelpFormatter)
    parser.add_argument('--server',
                        type=str,
                        default='localhost:9000',
                        help='PredictionService host:port')
    parser.add_argument('--model_name',
                        type=str,
                        default="my-model",
                        help='Name of the model')
    parser.add_argument('--input_image',
                        type=str,
                        default='./test_images/123.jpg',
                        help='Path to input image')
    parser.add_argument('--output_directory',
                        type=str,
                        default='./',
                        help='Path to output directory')
    parser.add_argument('--label_map',
                        type=str,
                        default="./data/object_detection.pbtxt",
                        help='Path to label map file')

    args = parser.parse_args()
    main(args)

I have used kubectl port forwarding for testing purposes so the request port is set to localhost:9000.

The output is :

prediction time : 6.690936326980591
[{b'goi_logo': 0.9999970197677612}]
Overall Time : 10.25893259048462

What can I do to make my inference faster? I have seen that the inference time is in the order of milliseconds so in comparison 10 seconds is a very long duration and unfit for production environments. I understand that port forwarding is slow. What is another method that I can use? I need to make this client available to the world as an API endpoint.

Answer 1

As previous answers stated, you should indeed try to do multiple requests because tf-serving needs some overhead the first time(s). You can prevent this by using a warm-up script.

To add some extra options:

• from tf-serving v1.8 you can also use a http rest API service. Then you can call the service that you have created on your GKE from a google compute engine to reduce the connection lag. In my case it had a big speed-up because my local connection was mediocre at best. Next to http rest api being more workable to debug, you can also send much bigger requests. The grpc limit seems to be 1.5 mb while the http one is a lot higher.

• Are you sending b64 encoded images? Sending the images themselves is a lot slower than sending b64 encoded strings. The way I handled this is sending b64 encoded strings from the images and create some extra layers in front of my network that transform the string to jpeg images again and then process them through the model. Some code to help you on your way:

from keras.applications.inception_v3 import InceptionV3, preprocess_input
from keras.models import Model
import numpy as np
import cv2
import tensorflow as tf
from keras.layers import Input, Lambda
from keras import backend as K

base_model = InceptionV3(
                weights='imagenet',
                include_top=True)

model = Model(
    inputs=base_model.input,
    outputs=base_model.get_layer('avg_pool').output)



def prepare_image(image_str_tensor):
            #image = tf.squeeze(tf.cast(image_str_tensor, tf.string), axis=[0])
            image_str_tensor = tf.cast(image_str_tensor, tf.string)
            image = tf.image.decode_jpeg(image_str_tensor,
                                        channels=3)
            #image = tf.divide(image, 255)
            #image = tf.expand_dims(image, 0)
            image = tf.image.convert_image_dtype(image, tf.float32)
            return image

def prepare_image_batch(image_str_tensor):
    return tf.map_fn(prepare_image, image_str_tensor, dtype=tf.float32)

# IF BYTE STR

model.layers.pop(0)
print(model.layers[0])

input_img = Input(dtype= tf.string,
            name ='string_input',
            shape = ()
            )
outputs = Lambda(prepare_image_batch)(input_img)
outputs = model(outputs)
inception_model = Model(input_img, outputs)
inception_model.compile(optimizer = "sgd", loss='categorical_crossentropy')
weights = inception_model.get_weights()

• Next to that, I would say use a bigger gpu. I have basic yolo (keras implementation) now running on a P100 with about 0.4s latency when called from a compute engine. We noticed that the darknet implementation (in c++) is a lot faster than the keras implementation tho.