Validation loss for pytorch Faster-RCNN

Question

I’m currently doing object detection on a custom dataset using transfer learning from a pytorch pretrained Faster-RCNN model (like in torchvision tutorial). I would like to compute validation loss dict (as in train mode) at the end of each epoch. I can just run model in train mode for validation like this:

model.train()
for images, targets in data_loader_val:
    images = [image.to(device) for image in images]
    targets = [{k: v.to(device) for k, v in t.items()} for t in targets]

    with torch.no_grad():
         val_loss_dict = model(images, targets)
         print(val_loss_dict)

but I don't think, that it's "correct" way to validate (cause some special layers like dropout and batch norm works different in eval/train mode). And in eval mode model returns predicted bboxes (as expected). Can I use some build-in function for this?

Thanks.

Answer 1

There was some discussion about this issue here. The conclusion there is that it is absolutely valid to calculate validation loss in train mode. The numerical value of the val loss in itself is not meaningful, only the trend is important to prevent overfitting. Therefore while train mode does alter the numerical value of the loss, it's still valid to use.

---

There is however another issue with efficiency here, in case you also need the model outputs in the validation process (for calculating IoU, accuracy, etc. as is often the case). Right now RCNN in torchvision gives you either losses or outputs, depending on training/eval mode.

UPDATE: I realized this fix is not working unfortunately. All submodules would have to be patched to calculate both losses and outputs. Too bad.

My dirty solution was patching the GeneralizedRCNN class from which FasterRCNN inherits. The problem is in this line, in eager_outputs(). The workaround:

    return losses, detections

model = fasterrcnn_resnet50_fpn() model.eager_outputs =
eager_outputs_patch

Now you can get both outputs after a single inference run: model.train() with torch.no_grad(): loss_dict, outputs = model(images, targets). # yaay, now we have both! Note that you still need to put your model to train mode in order to have the losses too. In eval mode GeneralizedRCNN's submodules (rpn, roi_heads) don't calculate any loss, and loss_dict is empty.

Answer 2

I have solved this problem by editing Generalized RCNN, RPN, roi_heads. Just add an if-statement to handle when targets are passed to still calculate loss even if not in training mode. For example in RPN:

losses = {}
    if self.training:
        assert targets is not None
        labels, matched_gt_boxes = self.assign_targets_to_anchors(anchors, targets)
        regression_targets = self.box_coder.encode(matched_gt_boxes, anchors)
        loss_objectness, loss_rpn_box_reg = self.compute_loss(
            objectness, pred_bbox_deltas, labels, regression_targets)
        losses = {
            "loss_objectness": loss_objectness,
            "loss_rpn_box_reg": loss_rpn_box_reg,
        }
    else:
        if targets is not None:
            labels, matched_gt_boxes = self.assign_targets_to_anchors(anchors, targets)
            regression_targets = self.box_coder.encode(matched_gt_boxes, anchors)
            loss_objectness, loss_rpn_box_reg = self.compute_loss(
                objectness, pred_bbox_deltas, labels, regression_targets)
            losses = {
                "loss_objectness": loss_objectness,
                "loss_rpn_box_reg": loss_rpn_box_reg,
            }