TensorFlow Object Detection API Weird Behavior

Question

I was playing with TensorFlow's brand new Object Detection API and decided to train it on some other publicly available datasets.

I happened to stumble upon this grocery dataset which consists of images of various brands of cigarette boxes on the supermarket shelf along with a text file which lists out the bounding boxes of each cigarette box in each image. 10 major brands have been labeled in the dataset and all other brands fall into the 11th "miscellaneous" category.

I followed their tutorial and managed to train the model on this dataset. Due to limitations on processing power, I used only a third of the dataset and performed a 70:30 split for training and testing data. I used the faster_rcnn_resnet101 model. All parameters in my config file are the same as the default parameters provided by TF.

After 16491 global steps, I tested the model on some images but I am not too happy with the results -

Failed to detect the Camels in top-shelf whereas it detects the product in other images

Why does it fail to detect the Marlboros in the top row?

Another issue I had is that the model never detected any other label except for label 1

Doesn't detected a crop instance of the product from the training data

It detects cigarette boxes with 99% confidence even in negative images!

Can somebody help me with what is going wrong? What can I do to improve the accuracy? And why does it detect all products to belong in category 1 even though I have mentioned that there are 11 classes in total?

Edit Added my label map:

item {   id: 1   name: '1' }  item {   id: 2   name: '2' }  item {   id: 3   name: '3' }  item {   id: 4   name: '4' }  item {   id: 5   name: '5' }  item {   id: 6   name: '6' }  item {   id: 7   name: '7' }  item {   id: 8   name: '8' }  item {   id: 9   name: '9' }  item {   id: 10   name: '10' }  item {   id: 11   name: '11' } 

Answer 1

So I think I figured out what is going on. I did some analysis on the dataset and found out that it is skewed towards objects of category 1.

This is the frequency distribution of the each category from 1 to 11 (in 0 based indexing)

0 10440 1 304 2 998 3 67 4 412 5 114 6 190 7 311 8 195 9 78 10 75 

I guess the model is hitting a local minima where just labelling everything as category 1 is good enough.

About the problem of not detecting some boxes : I tried training again, but this time I didn't differentiate between brands. Instead, I tried to teach the model what a cigarette box is. It still wasn't detecting all the boxes.

Then I decided to crop the input image and provide that as an input. Just to see if the results improve and it did!

It turns out that the dimensions of the input image were much larger than the 600 x 1024 that is accepted by the model. So, it was scaling down these images to 600 x 1024 which meant that the cigarette boxes were losing their details :)

So, I decided to test the original model which was trained on all classes on cropped images and it works like a charm :)

This was the output of the model on the original image

This is the output of the model when I crop out the top left quarter and provide it as input.

Thanks everyone who helped! And congrats to the TensorFlow team for an amazing job for the API :) Now everybody can train object detection models!

Answer 2

How many images are there in the dataset? The more training data that you have the better the API performs. I tried training it on about 20 images per class, the accuracy was pretty bad. I pretty much faced all the problems that you have mentioned above. When I generated more data, the accuracy improved considerably.

PS: Sorry I couldn't comment since I don't have enough reputation