When should I stop the object detection model training while mAP are not stable?

Question

I am re-training the SSD MobileNet with 900 images from the Berkeley Deep Drive dataset, and eval towards 100 images from that dataset.

The problem is that after about 24 hours of training, the totalloss seems unable to go below 2.0:

And the corresponding mAP score is quite unstable:

In fact, I have actually tried to train for about 48 hours, and the TotoalLoss just cannot go below 2.0, something ranging from 2.5~3.0. And during that time, mAP is even lower..

So here is my question, given my situation (I really don't need any "high-precision" model, as you can see, I pick 900 images for training and would like to simply do a PoC model training/predication and that's it), when should I stop the training and obtain a reasonably performed model?

Answer 1

indeed for detection you need to finetune the network, since you are using SSD, there are already some sources out there:

• https://gluon-cv.mxnet.io/build/examples_detection/finetune_detection.html (This one specifically for an SSD Model, uses mxnet but you can use the same with TF)
• You can watch a very nice finetuning intro here
• This repo has a nice fine tuning option enabled as long as you write your dataloader, check it out here

In general your error can be attributed to many factors, the learning rate you are using, the characteristics of the images themselves (are they normalized?) If the ssd network you are using was trained with normalized data and you don't normalize to retrain then you'll get stuck while learning. Also what learning rate are they using?

From the model zoo I can see that for SSD there are models trained on COCO

And models trained on Open Images:

If for example you are using ssd_inception_v2_coco, there is a truncated_normal_initializer in the input layers, so take that into consideration, also make sure the input sizes are the same that the ones you provide to the model.

You can get very good detections even with little data if you also include many augmentations and take into account the rest of the things I mentioned, more details on your code would help to see where the problem lies.