TensorFlow: How to handle void labeled data in image segmentation?

Question

I was wondering how to handle not labeled parts of an image in image segmentation using TensorFlow. For example, my input is an image of height * width * channels. The labels are too of the size height * width, with one label for every pixel.

Some parts of the image are annotated, other parts are not. I would wish that those parts have no influence on the gradient computation whatsoever. Furthermore, I am not interested in the network predicting this “void” label.

Is there a label or a function for this? At the moment I am using tf.nn.sparse_softmax_cross_entropy_with_logits.

Answer 1

I'm not 100% familiar with TF. However, have you considered using the weights parameter of the loss?
Looking at tf.loses.sparse_softmax_cross_entropy it has a parameter weights

weights: Coefficients for the loss. This must be scalar or of same rank as labels

You can set weightof "void" pixels to zero, thus making the loss ignore them.

You can also remove the reduction from tf.nn.sparse_softmax_cross_entropy_with_logits and use tf.losses.compute_weighted_loss to perform the weighting.

Answer 2

If I understand correctly you have a portion of each image with label void in which you are not interested at all. Since there is not a easy way to obtain the real value behind this void spots, why don't you map these points to background label and try to get results for your model? I would try in a preprocessing state to clear the data labels from this void label and substitute them with background label.

Another possible strategy ,if you don's simply want to map void labels to background, is to run a mask (with a continuous motion from top to bottom from right to left) to check the neigthbooring pixels from a void pixel (let's say an area of 5x5 pixels) and assign to the void pixels the most common label besides void.

Also you can always keep a better subset of the data, filtering data where the percentage of void labels is over a threshold. You can keep only images with no void labels, or more likeley you can keep images that have only under a threshold (e.g. 5%) of non-labeled points. In this images you can implement the beforementioned strategies for replacing the void labels.