How to debug a custom loss function?

Question

I don't have a specific problem at the moment, but it keeps coming up that I have a bug in my loss function and the error printouts are not sufficient to localize the problem to a specific line of code. For example expected 'int32' but got 'float32' or something like that. Is there a way to know which line of code in the loss function is the source of the problem?

I'll note that sometimes the error comes during compilation, in which case print statements have been helpful. But I have not identified a way to find the problem (outside of guessing or commenting out sections) if it happens only during training, since printouts are not displayed.

Answer 1

You can use one of following option to debug custom loss function or any other deep learning works.

1. tf.print
2. tfbdg
3. TensorBoard debugger