TensorFlow: Adding Class to Pre-trained Inception Model & Outputting Full Image Hierarchy

Question

Two questions:

1) Does anyone know if I can add new image classes to the pre-trained Inception-v3 model? For example, I wanted to train TensorFlow on a multitude of national flags, but I need to make sure that I can still recognize the images from the ImageNet hierarchy. I realize that there is a way to wipe the top layer of Inception and completely retrain the model on my classes, but this very limiting and time consuming.

2) Also, is there a way to output the entire hierarchy containing the tag that the image receives? I wish to be able to not only see specifically what Inception tags the image as, but I want to see all of the more broad 'synsets' from ImageNet. For example, instead of just seeing the output "toy poodle", I am interested in "Animal/Domesticated Animal/Dog/Poodle/toy poodle".

Any responses are greatly appreciated.

Answer 1

1) Output layer is softmax, which means it has predefined number of neurons, each one is defined for one specific class. Technically you could perform Network Surgery so that it has one more neuron in output layer which will represent your new class. But you will have to perform additional training of your network so that it updates all of its weights in order to account for new class. Bad news - it could take a while since update will affect whole network and the network is GIANT. Good news - such change in pretrained existing network will be faster than learning everything from scratch.

2) What makes you think that such hierarchy exists at all? You can't know anything about internal representation of data for sure. Of course, you can inspect activations of neurons in each of the functions and even visualize them... But you will have to try to understand what these activations mean on your own. And probably you won't find any hierarchy which you expect to see. So to sum up - understanding how ANN represents data internally is no easy task. Actually - extremely difficult one.

Suggested further reading: https://github.com/tensorflow/models/tree/master/inception

Pay attention to this part of the doc - it is strongly related to your #1

Answer 2

1. Here is related question: How to get existing model to recognize additional classes?

Here is some explanations: https://github.com/tensorflow/models/issues/2510.

So it is possible somehow finetune model if having model checkpoint. Here is repo link with example of finetuning: https://github.com/tensorflow/models/tree/master/research/slim/

2. It is possible to get node name and human readable slug for class, so you can do manual linking of nodes by categories. But it is time consuming.