Image similarity detection with TensorFlow

Question

Recently I started to play with tensorflow, while trying to learn the popular algorithms i am in a situation where i need to find similarity between images.

Image A is supplied to the system by me, and userx supplies an image B and the system should retrieve image A to the userx if image B is similar(color and class).

Now i have got few questions:

1. Do we consider this scenario to be supervised learning? I am asking because i don't see it as a classification problem(confused!!)
2. What algorithms i should use to train etc..
3. Re-training should be done quite often, how should i tackle this problem so i don't train everytime from scratch( fine-tuning??)

Answer 1

Do we consider this scenario to be supervised learning?

It is supervised learning when you have labels to optimize your model. So for most neural networks, it is supervised.

However, you might also look at the complete task. I guess you don't have any ground truth for image pairs and the "desired" similarity value your model should output?

One way to solve this problem which sounds inherently unsupervised is to take a CNN (convolutional neural network) trained (in a supervised way) on the 1000 classes of image net. To get the similarity of two images, you could then simply take the euclidean distance of the output probability distribution. This will not lead to excellent results, but is probably a good starter.

2. What algorithms i should use to train etc..

First, you should define what "similar" means for you. Are two images similar when they contain the same object (classes)? Are they similar if the general color of the image is the same?

For example, how similar are the following 3 pairs of images?

Have a look at FaceNet and search for "Content based image retrieval" (CBIR):

• Wikipedia
• Google Scholar

Answer 2

1. This can be a supervised learning. You can classify the images into categories, if two images are in the same categories (or close in a category), you can think of them as similar.

2. You can use the deep conventional neural networks for imagenet such as inception model. The inception model outputs a probability map for 1000 classes (which is a vector whose values sum to 1). You can calculate the distance of vectors of two images to get their similarity.

3. On the same page of the inception model, you will also find the instructions to retrain a model: https://github.com/tensorflow/models/tree/master/inception#how-to-fine-tune-a-pre-trained-model-on-a-new-task