MInimum working example tensorflow serving client

Question

I am working on the basic Tensorflow Serving example. I am following the MNIST example, except instead of classification I want to use a numpy array to predict another numpy array.

To do this I first trained my neural network

x = tf.placeholder("float", [None, n_input],name ="input_values")

weights = {
    'encoder_h1': tf.Variable(tf.random_normal([n_input, n_hidden_1])),
    'encoder_h2': tf.Variable(tf.random_normal([n_hidden_1, n_hidden_2])),
    'encoder_h3': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_3])),
    'decoder_h1': tf.Variable(tf.random_normal([n_hidden_3, n_hidden_2])),
    'decoder_h2': tf.Variable(tf.random_normal([n_hidden_2, n_hidden_1])),
    'decoder_h3': tf.Variable(tf.random_normal([n_hidden_1, n_input])),
}
biases = {
    'encoder_b1': tf.Variable(tf.random_normal([n_hidden_1])),
    'encoder_b2': tf.Variable(tf.random_normal([n_hidden_2])),
    'encoder_b3': tf.Variable(tf.random_normal([n_hidden_3])),
    'decoder_b1': tf.Variable(tf.random_normal([n_hidden_2])),
    'decoder_b2': tf.Variable(tf.random_normal([n_hidden_1])),
    'decoder_b3': tf.Variable(tf.random_normal([n_input])),
}

# Building the encoder
def encoder(x):
    # Encoder Hidden layer with sigmoid activation #1
    layer_1 = tf.nn.tanh(tf.matmul(x, weights['encoder_h1'])+biases['encoder_b1'])
    print(layer_1.shape)
    # Decoder Hidden layer with sigmoid activation #2
    layer_2 = tf.nn.tanh(tf.matmul(layer_1, weights['encoder_h2'])+biases['encoder_b2'])
    print(layer_2.shape)
    # Layer 3
    layer_3 = tf.nn.tanh(tf.matmul(layer_2, weights['encoder_h3'])+biases['encoder_b3'])
    print(layer_3.shape)
    return layer_3


# Building the decoder
def decoder(x):
    # Encoder Hidden layer with sigmoid activation #1
    layer_1 = tf.nn.tanh(tf.matmul(x, weights['decoder_h1'])+biases['decoder_b1'])
    print(layer_1.shape)
    # Decoder Hidden layer with sigmoid activation #2
    layer_2 = tf.nn.tanh(tf.matmul(layer_1, weights['decoder_h2'])+biases['decoder_b2'])
    # Layer 3
    layer_3 = tf.nn.tanh(tf.matmul(layer_2, weights['decoder_h3'])+biases['decoder_b3'])
    return layer_3

# Construct model
encoder_op = encoder(x)
decoder_op = decoder(encoder_op)

# Prediction
y = decoder_op



# Objective functions
y_ = tf.placeholder("float", [None,n_input],name="predict")

Next as someone suggested here I saved have my network like so..

import os
import sys

from tensorflow.python.saved_model import builder as saved_model_builder
from tensorflow.python.saved_model import utils
from tensorflow.python.saved_model import tag_constants, signature_constants
from tensorflow.python.saved_model.signature_def_utils_impl import     build_signature_def, predict_signature_def
from tensorflow.contrib.session_bundle import exporter

with tf.Session() as sess:
# Initialize variables
    sess.run(init)

    # Restore model weights from previously saved model
    saver.restore(sess, model_path)
    print("Model restored from file: %s" % save_path)

    export_path = '/tmp/AE_model/6'
    print('Exporting trained model to', export_path)
    builder = tf.saved_model.builder.SavedModelBuilder(export_path)


    signature = predict_signature_def(inputs={'inputs': x},
                                  outputs={'outputs': y})

    builder.add_meta_graph_and_variables(sess=sess,
                                         tags=[tag_constants.SERVING],
                                         signature_def_map={'predict': signature})

    builder.save()


    print 'Done exporting!'

Next I follow the instructions to run my server on localhost:9000

bazel build //tensorflow_serving/model_servers:tensorflow_model_server

I set up the server

bazel-bin/tensorflow_serving/model_servers/tensorflow_model_server --port=9000 --model_base_path=/tmp/AE_model/

THE PROBLEM

Now I want to write a program so I can pass Mat vectors from a C++ program in eclipse(I use a LOT of libraries) to my server so I can make some sort of predictions.

I first thought about using the inception_client.cc as reference. However, it seems that I need Bazel to compile it because I cannot find prediction_service.grpc.pb.h anywhere :(

so it seems that my only other option is to call a script using python I get the following output:

<grpc.beta._client_adaptations._Rendezvous object at 0x7f9bcf8cb850>

Any help with this problem would be appreciated.

Thank you.

EDIT:

I reinstalled protobuf and grpc and ran the commands as suggested:

My command was a little different and I had to use it outside my serving folder(In Ubuntu 14.04).

sudo protoc -I=serving -I serving/tensorflow --grpc_out=. --plugin=protoc-gen-grpc=`which grpc_cpp_plugin` serving/tensorflow_serving/apis/*.proto

This generated the .gprc.pb.h files and I dragged them into the /apis/ folder and the errors went down. Now I get the error

/tensorflow/third_party/eigen3/unsupported/Eigen/CXX11/Tensor:1:42: fatal error: unsupported/Eigen/CXX11/Tensor: No such file or directory

Even though this file does exist. Any suggestions are appreciated.

Thank you @subzero!

EDIT 2

I was able to solve the problem with Eigen by updating to the newest Eigen version and building from source. Next I pointed to the /usr/local/include/eigen3/

Afterwards I had issues with the tensorflow libraries. These I resolved by generating the libtensorflow_cc.so library using lababidi's suggestion. https://github.com/tensorflow/tensorflow/issues/2412

I have one last issue. Everything seems to be fine except I get the error:

undefined reference to`tensorflow::serving::PredictRequest::~PredictRequest()'

It seems that I am missing either a linker or a library. Does anyone know what I am missing???

Answer 1

An example of a custom client and server:

server code to add to a tensorflow model:

import grpc
from concurrent import futures
import python_pb2
import python_pb2_grpc

class PythonServicer(python_pb2_grpc.PythonServicer):


    def makePredictions(self, request, context):


        items = eval(str(request.items)) #Receives the input values for the model as a string and evaluates them into an array to be passed to tensorflow

        x_feed = items

        targetEval_out = sess.run(confidences, feed_dict={x:x_feed}) #"confidences" is the output of my model, replace it for the appropriate function from your model


        out = str(targetEval_out.tolist()) #The model output is then put into string format to be passed back to the client. It has to be reformatted on the other end, but this method was easier to implement

        return python_pb2.value(name=out)


print("server online")
MAX_MESSAGE_LENGTH = 4 * 1024 * 1024 #can be edited to allow for larger amount of data to be transmitted per message. This can be helpful for making large numbers of predictions at once.
server = grpc.server(futures.ThreadPoolExecutor(max_workers=10), 
options=[('grpc.max_send_message_length', MAX_MESSAGE_LENGTH), (
'grpc.max_receive_message_length', MAX_MESSAGE_LENGTH)])
python_pb2_grpc.add_PythonServicer_to_server(
PythonServicer(), server)
server.add_insecure_port('[::]:50051')
server.start()

Client C++ code:

#include <grpc/grpc.h>
#include <grpc++/channel.h>
#include <grpc++/client_context.h>
#include <grpc++/create_channel.h>
#include <grpc++/security/credentials.h>
#include "python.grpc.pb.h"

using grpc::Channel;
using grpc::ClientContext;
using grpc::ClientReader;
using grpc::ClientReaderWriter;
using grpc::ClientWriter;
using grpc::Status;
using python::request;
using python::value;
using python::Python;

using namespace std;


unsigned MAX_MESSAGE_LENGTH = 4 * 1024 * 1024; //can be edited to allow for larger amount of data to be transmitted per message. This can be helpful for making large numbers of predictions at once.
grpc::ChannelArguments channel_args;
channel_args.SetMaxReceiveMessageSize(MAX_MESSAGE_LENGTH);
channel_args.SetMaxSendMessageSize(MAX_MESSAGE_LENGTH);

shared_ptr<Channel> channel = CreateCustomChannel("localhost:50051", grpc::InsecureChannelCredentials(),channel_args);
unique_ptr<python::Python::Stub>stub = python::Python::NewStub(channel);

request r;
r.set_items(dataInputString); //The input data should be a string that can be parsed to a python array, for example "[[1.0,2.0,3.0],[4.0,5.0,6.0]]"
//The server code was made to be able to make multiple predictions at once, hence the multiple data arrays 
value val;
ClientContext context;

Status status = stub->makePredictions(&context, r, &val);

cout << val.name() << "\n"; //This prints the returned model prediction

The python.proto code:

syntax = "proto3";


package python;

service Python {

    rpc makePredictions(request) returns (value) {}


}

message request {
  string items = 1;
}


message value {
  string name = 1;
}

I'm not sure whether these code snippets work on their own, as I just copied the relevant code from my current project. But hopefully this will help as a good starting place for anyone needing a tensorflow client and server.