How to structure Tensorflow model code?

Question

I am having difficulty finding how to structure my Tensorflow model code. I would like to structure it in the form of a Class for ease for future reuse. Also, my current structure is messy and the tensorboard graph output have multiple "model" inside.

The following is what i have currently:

import tensorflow as tf
import os

from utils import Utils as utils

os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'

class Neural_Network:
    # Neural Network Setup
    num_of_epoch = 50

    n_nodes_hl1 = 500
    n_nodes_hl2 = 500
    n_nodes_hl3 = 500

    def __init__(self):
        self.num_of_classes = utils.get_num_of_classes()
        self.num_of_words = utils.get_num_of_words()

        # placeholders
        self.x = tf.placeholder(tf.float32, [None, self.num_of_words])
        self.y = tf.placeholder(tf.int32, [None, self.num_of_classes])

        with tf.name_scope("model"):
            self.h1_layer = tf.layers.dense(self.x, self.n_nodes_hl1, activation=tf.nn.relu, name="h1")
            self.h2_layer = tf.layers.dense(self.h1_layer, self.n_nodes_hl2, activation=tf.nn.relu, name="h2")
            self.h3_layer = tf.layers.dense(self.h2_layer, self.n_nodes_hl3, activation=tf.nn.relu, name="h3")

            self.logits = tf.layers.dense(self.h3_layer, self.num_of_classes, name="output")

    def predict(self):
        return self.logits

    def make_prediction(self, query):
        result = None

        with tf.Session() as sess:
            saver = tf.train.import_meta_graph('saved_models/testing.meta')
            saver.restore(sess, 'saved_models/testing')

            sess.run(tf.global_variables_initializer())

            prediction = self.predict()
            prediction = sess.run(prediction, feed_dict={self.x : query})
            prediction = prediction.tolist()
            prediction = tf.nn.softmax(prediction)
            prediction = sess.run(prediction)
            print prediction

            return utils.get_label_from_encoding(prediction[0])

    def train(self, data):

        print len(data['values'])
        print len(data['labels'])

        prediction = self.predict()

        cost = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=prediction, labels=self.y))
        optimizer = tf.train.AdamOptimizer().minimize(cost)

        with tf.Session() as sess:
            sess.run(tf.global_variables_initializer())

            writer = tf.summary.FileWriter("mygraph/logs", tf.get_default_graph())

            for epoch in range(self.num_of_epoch):
                optimised, loss = sess.run([optimizer, cost],
                                           feed_dict={self.x: data['values'], self.y: data['labels']})

                if epoch % 1 == 0:
                    print("Completed Training Cycle: " + str(epoch) + " out of " + str(self.num_of_epoch))
                    print("Current Loss: " + str(loss))

                    saver = tf.train.Saver()
                    saver.save(sess, 'saved_models/testing')
                    print("Model saved")

What i have found online is that many use much lower level code such as tf.Variables and tf.Constant hence, they are much more able to split up their code. However, as I am relatively new to Tensorflow, I would like to use higher level code first.

Can anyone advise me on how to structure my code?

Answer 1

As commented, a short answer to your initial question would be to read this, but as you asked a follow-up question in your comments, I thought it needs a more complete answer.

---

Can anyone advise me on how to structure my code?

Obviously, structuring your code is a matter of taste. But, to help you making your own taste, here is the main thing you need to keep in mind: there are 2 different layers in TensorFlow, don't confuse them.

• The first one is the Graph layer which holds all your TensorFlow nodes such as
  • tensors (e.g. tf.placeholder, tf.constant, tf.Variables, etc..), or
  • operations (tf.add, tf.matmul, etc..). The Graph contains your model per se and might contains much more such as: a loss function, an optimizer to train your model, an input data pipeline, etc.

Each of this nodes has a name you can use to retrieve it directly from the graph (e.g. using the tf.get_variable method or tf.Graph.get_tensor_by_name).

• The second layer is the way you build your TensorFlow Graph using the Python (or the C++ or the Java, ...) API. This is likely this layer you had in mind when asking your question. But, in a way, this layer is really more a model factory rather than a model.

---

Does the format support saving and restoring of models?

It depends on what you mean by model even if the answer is yes in both cases.

• If you had in mind the TensorFlow Graph, the answer is yes, you can save and restore your Graph because it doesn't depend on how you construct it. Just look at the Saving and Restoring section of this document to get some insights on how to do it or look at this answer where only the Graph is restored

• If you had in mind the Python class, the short answer is no. But you can make up something to make it a yes.
  As written in the previous item, TensorFlow checkpoint doesn't saved the Python (nor the C++ or Java) object, but only the graph. But the structure of your model as a Python class lives somewhere else: it lives in your code.

  So if you recreate an instance of your Python class, and if you make sure that all your TensorFlow nodes are recreated in the Graph (so you get an equivalent Graph), then, when you'll restore the TensorFlow Graph from the checkpoint, your model as a Python-instance-linked-to-a-TensorFlow-Graph will have been restored.

  See the Restoring Variables section of document for a trivial example where the Python-instances-linked-to-a-TensorFlow-Graph are actually Python variables (namely v1 and v2) living somewhere in the module scope.

  # Create some variables.
  v1 = tf.Variable(..., name="v1")
  v2 = tf.Variable(..., name="v2")
  ...
  
  # Add ops to save and restore all the variables.
  saver = tf.train.Saver()
  
  # Later, launch the model, use the saver to restore variables from disk, and
  # do some work with the model.
  with tf.Session() as sess:
    # Restore variables from disk.
    saver.restore(sess, "/tmp/model.ckpt")
    print("Model restored.")
    # Do some work with the model
    ...
  

---

I can only recommend to read (and upvote :)) this question and its answers as you will learn a lot in terms of how saving/restoring work in TensorFlow.

---

Hopefully, it's a bit clearer now.