Inference error with TensorFlow C++ on iOS: "Invalid argument: Session was not created with a graph before Run()!"

Question

I am trying to run my model on iOS using TensorFlow's C++ API. The model is a SavedModel saved as a .pb file. However, calls to Session::Run() result in the error:

"Invalid argument: Session was not created with a graph before Run()!"

In Python, I can successfully run inference on the model with the following code:

with tf.Session() as sess:
    tf.saved_model.loader.load(sess, ['serve'], '/path/to/model/export')
    result = sess.run(['OutputTensorA:0', 'OutputTensorB:0'], feed_dict={
        'InputTensorA:0': np.array([5000.00] * 1000).reshape(1, 1000),
        'InputTensorB:0': np.array([300.00] * 1000).reshape(1, 1000)
    })
    print(result[0])
    print(result[1])

In C++ on iOS, I try to mimick this working snippit as follows:

tensorflow::Input::Initializer input_a(5000.00, tensorflow::TensorShape({1, 1000}));
tensorflow::Input::Initializer input_b(300.00, tensorflow::TensorShape({1, 1000}));

tensorflow::Session* session_pointer = nullptr;

tensorflow::SessionOptions options;
tensorflow::Status session_status = tensorflow::NewSession(options, &session_pointer);

std::cout << session_status.ToString() << std::endl; // prints OK

std::unique_ptr<tensorflow::Session> session(session_pointer);

tensorflow::GraphDef model_graph;

NSString* model_path = FilePathForResourceName(@"saved_model", @"pb");
PortableReadFileToProto([model_path UTF8String], &model_graph);

tensorflow::Status session_init = session->Create(model_graph);

std::cout << session_init.ToString() << std::endl; // prints OK

std::vector<tensorflow::Tensor> outputs;
tensorflow::Status session_run = session->Run({{"InputTensorA:0", input_a.tensor}, {"InputTensorB:0", input_b.tensor}}, {"OutputTensorA:0", "OutputTensorB:0"}, {}, &outputs);

std::cout << session_run.ToString() << std::endl; // Invalid argument: Session was not created with a graph before Run()!

The methods FilePathForResourceName and PortableReadFileToProto are taken from the TensorFlow iOS sample found here.

What is the problem? I noticed that this happens regardless of how simple the model is (see my issue report on GitHub), which means the problem is not with the specifics of the model.

Answer 1

The primary issue here is that you are exporting your graph to a SavedModel in Python but then reading it in as a GraphDef in C++. While both have a .pb extension and are similar, they are not equivalent.

What is happening is you are reading in the SavedModel with PortableReadFileToProto() and it is failing, leaving an empty pointer (model_graph) to a GraphDef object. So after the execution of PortableReadFileToProto(), model_graph remains an empty, but valid, GraphDef, which is why the error says Session was not created with a graph before Run(). session->Create() succeeds because you successfully created a session with an empty graph.

The way to check if PortableReadFileToProto() fails is to check its return value. It returns a bool, which will be 0 if reading in the graph failed. If you wish to obtain a descriptive error here, use ReadBinaryProto(). Another way you can tell if reading the graph failed is by checking the value of model_graph.node_size(). If this is 0, then you have an empty graph and reading it in has failed.

While you can use TensorFlow's C API to perform inference on a SavedModel by using TF_LoadSessionFromSavedModel() and TF_SessionRun(), the recomended method is to export your graph to a frozen model using freeze_graph.py or write to a GraphDef using tf.train.write_graph(). I will demonstrate successful inference with a model exported using tf.train.write_graph():

In Python:

# Build graph, call it g
g = tf.Graph()

with g.as_default():
    input_tensor_a = tf.placeholder(dtype=tf.int32, name="InputTensorA")
    input_tensor_b = tf.placeholder(dtype=tf.int32, name="InputTensorB")
    output_tensor_a = tf.stack([input_tensor_a], name="OutputTensorA")
    output_tensor_b = tf.stack([input_tensor_b], name="OutputTensorB")

# Save graph g
with tf.Session(graph=g) as sess:
    sess.run(tf.global_variables_initializer())
    tf.train.write_graph(
        graph_or_graph_def=sess.graph_def,
        logdir='/path/to/export',
        name='saved_model.pb',
        as_text=False
    )

In C++ (Xcode):

using namespace tensorflow;
using namespace std;

NSMutableArray* predictions = [NSMutableArray array];

Input::Initializer input_tensor_a(1, TensorShape({1}));
Input::Initializer input_tensor_b(2, TensorShape({1}));

SessionOptions options;
Session* session_pointer = nullptr;
Status session_status = NewSession(options, &session_pointer);
unique_ptr<Session> session(session_pointer);

GraphDef model_graph;
string model_path = string([FilePathForResourceName(@"saved_model", @"pb") UTF8String]);

Status load_graph = ReadBinaryProto(Env::Default(), model_path, &model_graph);

Status session_init = session->Create(model_graph);

cout << "Session creation Status: " << session_init.ToString() << endl;
cout << "Number of nodes in model_graph: " << model_graph.node_size() << endl;
cout << "Load graph Status: " << load_graph.ToString() << endl;

vector<pair<string, Tensor>> feed_dict = {
    {"InputTensorA:0", input_tensor_a.tensor},
    {"InputTensorB:0", input_tensor_b.tensor}
};

vector<Tensor> outputs;
Status session_run = session->Run(feed_dict, {"OutputTensorA:0", "OutputTensorB:0"}, {}, &outputs);

[predictions addObject:outputs[0].scalar<int>()];
[predictions addObject:outputs[1].scalar<int>()];

Status session_close = session->Close();

This general method will work, but you will likely experience issues with required operations missing from the TensorFlow library you built and therefore inference would still fail. To combat this, first make sure that you have built the latest TensorFlow 1.3 by cloning the repo on your machine and running tensorflow/contrib/makefile/build_all_ios.sh from the root tensorflow-1.3.0 directory. It is unlikely that inference will work for a custom, non-canned model if you use the TensorFlow-experimental Pod like the examples. Once you have a static library built using build_all_ios.sh, you need to link it up in your .xcconfig by following the instructions here.

Once you successfully link the static library built using the makefile with Xcode, you will likely still get errors that prevent inference. While the actual errors you will get depend on your implementation, you will receive errors that fall into two different forms:

1. OpKernel ('op: "[operation]" device_type: "CPU"') for unknown op: [operation]

2. No OpKernel was registered to support Op '[operation]' with these attrs. Registered devices: [CPU], Registered kernels: [...]

Error #1 means that the .cc file from tensorflow/core/ops or tensorflow/core/kernels for the corresponding operation (or closely associated operation) is not in the tf_op_files.txt file in tensorflow/contrib/makefile. You will have to find the .cc that contains REGISTER_OP("YourOperation") and add it to tf_op_files.txt. You must rebuild by running tensorflow/contrib/makefile/build_all_ios.sh again.

Error #2 means that the .cc file for the corresponding operation is in your tf_op_files.txt file, but you have supplied the operation with a data type that it (a) doesn't support or (b) is stripped off to reduce the size of the build.

One "gotcha" is that if you are using tf.float64 in the implementation of your model, this is exported as TF_DOUBLE in your .pb file and this is not supported by most operations. Use tf.float32 in place of tf.float64 and then re-save your model using tf.train.write_graph().

If you are still receiving error #2 after checking you are providing the correct datatype to the operation, you will need to either remove __ANDROID_TYPES_SLIM__ in the makefile located at tensorflow/contrib/makefile or replace it with __ANDROID_TYPES_FULL__ and then rebuild.

After getting passed errors #1 and #2, you will likely have successful inference.