Tensorflow serving retrained inception

Question

I am trying to serve my retrained inception model following this guide (you may also see this guide, which explains how to retrain inception). I've modified retrain.py to export my model as follows:

... # Same as in the original script:
# Set up the pre-trained graph.
maybe_download_and_extract()
graph, bottleneck_tensor, jpeg_data_tensor, resized_image_tensor = (create_inception_graph())
... # Same as in the original script:
# Add the new layer that we'll be training.
(train_step, cross_entropy, bottleneck_input, ground_truth_input, final_tensor) = add_final_training_ops(len(image_lists.keys()),
                                         FLAGS.final_tensor_name,
                                         bottleneck_tensor)
... # Added at the end of the original script:
# Export model
with graph.as_default():
    export_path = sys.argv[-1]
    print('Exporting trained model to', export_path)
    saver = tf.train.Saver(sharded=True)
    model_exporter = exporter.Exporter(saver)
    signature = exporter.classification_signature(input_tensor=jpeg_data_tensor, scores_tensor=final_tensor)
    model_exporter.init(sess.graph.as_graph_def(), default_graph_signature=signature)
    model_exporter.export(export_path, tf.constant(FLAGS.export_version), sess)
    print('Done exporting!')

if __name__ == '__main__':
  tf.app.run()

After exporting my model I start running the server:

/serving/bazel-bin/tensorflow_serving/example/inception_inference --port=9000 EXPORT_DIR &> inception_log &

Server log file (inception_log) contains:

I tensorflow_serving/core/basic_manager.cc:190] Using InlineExecutor for BasicManager.
I tensorflow_serving/example/inception_inference.cc:384] Waiting for models to be loaded...
I tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:147] File-system polling found servable version {name: default version: 1} at path /tf_files/scope/export/00000001
I external/org_tensorflow/tensorflow/contrib/session_bundle/session_bundle.cc:129] Attempting to load a SessionBundle from: /tf_files/scope/export/00000001
I tensorflow_serving/example/inception_inference.cc:384] Waiting for models to be loaded...
I tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:147] File-system polling found servable version {name: default version: 1} at path /tf_files/scope/export/00000001
I external/org_tensorflow/tensorflow/contrib/session_bundle/session_bundle.cc:106] Running restore op for SessionBundle
I external/org_tensorflow/tensorflow/contrib/session_bundle/session_bundle.cc:203] Done loading SessionBundle
I tensorflow_serving/example/inception_inference.cc:350] Running...
I tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:147] File-system polling found servable version {name: default version: 1} at path /tf_files/scope/export/00000001
I tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:147] File-system polling found servable version {name: default version: 1} at path /tf_files/scope/export/00000001
I tensorflow_serving/sources/storage_path/file_system_storage_path_source.cc:147] File-system polling found servable version {name: default version: 1} at path /tf_files/scope/export/00000001
... 

Finally, I run the client and I get the following error:

/serving/bazel-bin/tensorflow_serving/example/inception_client --server=localhost:9000 --image=TEST_IMG
D0805 09:10:46.208704633     200 ev_posix.c:101]             Using polling engine: poll
Traceback (most recent call last):
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/tensorflow_serving/example/inception_client.py", line 53, in <module>
    tf.app.run()
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/external/org_tensorflow/tensorflow/python/platform/app.py", line 30, in run
    sys.exit(main(sys.argv))
  File "/serving/bazel-bin/tensorflow_serving/example/inception_client.runfiles/tensorflow_serving/example/inception_client.py", line 48, in main
    result = stub.Classify(request, 10.0)  # 10 secs timeout
  File "/usr/local/lib/python2.7/dist-packages/grpc/beta/_client_adaptations.py", line 300, in __call__
    self._request_serializer, self._response_deserializer)
  File "/usr/local/lib/python2.7/dist-packages/grpc/beta/_client_adaptations.py", line 198, in _blocking_unary_unary
    raise _abortion_error(rpc_error_call)
    grpc.framework.interfaces.face.face.AbortionError: AbortionError(code=StatusCode.INTERNAL, details="FetchOutputs node : not found")
E0805 09:10:47.129263239     200 chttp2_transport.c:1810]    close_transport: {"created":"@1470388247.129230608","description":"FD shutdown","file":"src/core/lib/iomgr/ev_poll_posix.c","file_line":427}

Any advice or guidance in this matter would be greatly appreciated.

Answer 1

So the link on the tensorflow website is just one way to fully serve the model from my experience. A better way to serve the model would be to serve it up from Flask and Kubernetes instead, since its a lighter weight client than all of the tensorflow serving infrastructure, but this is assuming that the volume you have isn't very large(>100 QPS) although you could serve Inception up with Flask and Kubernetes with that load, but at that rate I would op for the in-line solution.

You could serve it from a remote service, and that would work, but depending on your infrastructure you "could" also serve that model up in a streaming job that pushed your requests through an apache_beam.DoFn then outputting it out back to a MQ that your job was listening to. This is just another solution. Hope this helps.

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import logging 
import tensorflow as tf
import numpy as np
import apache_beam as beam


class InferenceFn(beam.DoFn):

  def __init__(self, model_dict):
    super(InferenceFn, self).__init__()
    self.model_dict = model_dict
    self.graph = None
    self.create_graph()


  def create_graph(self):
    if not tf.gfile.FastGFile(self.model_dict['model_full_path']):
      self.download_model_file()
    with tf.Graph().as_default() as graph:
      with tf.gfile.FastGFile(self.model_dict['model_full_path'], 'rb') as f:
        graph_def = tf.GraphDef()
        graph_def.ParseFromString(f.read())
        _ = tf.import_graph_def(graph_def, name='')
    self.graph = graph

  def start_bundle(self):
    """Prevents graph object serialization until serving. Required for GCP Serving"""
    self.create_graph()

  def process(self, element):
    """Core Processing Fn for Apache Beam."""
    try:
      with tf.Session(graph=self.graph) as sess:
        if not tf.gfile.Exists(element):
          tf.logging.fatal('File does not exist %s', element)
          raise ReferenceError("Couldnt Find the image {}".format(element))
        data = tf.gfile.FastGFile(element, 'rb').read()
        output_tensor = sess.graph.get_tensor_by_name(self.model_dict['output_tensor_name'])
        predictions = sess.run(softmax_tensor, {self.model_dict['input_tensor_name']: data})
        predictions = np.squeeze(predictions)
        yield str(predictions)
    except Exception:
      logging.error("We hit an error in inference on {}".format(element))