Keras Implementation of Customized Loss Function that need internal layer output as label

Question

in keras, I want to customize my loss function which not only takes (y_true, y_pred) as input but also need to use the output from the internal layer of the network as the label for an output layer.This picture shows the Network Layout

Here, the internal output is xn, which is a 1D feature vector. in the upper right corner, the output is xn', which is the prediction of xn. In other words, xn is the label for xn'.

While [Ax, Ay] is traditionally known as y_true, and [Ax',Ay'] is y_pred.

I want to combine these two loss components into one and train the network jointly.

Any ideas or thoughts are much appreciated!

Answer 1

I have figured out a way out, in case anyone is searching for the same, I posted here (based on the network given in this post):

The idea is to define the customized loss function and use it as the output of the network. (Notation: A is the true label of variable A, and A' is the predicted value of variable A)

def customized_loss(args):
    #A is from the training data
    #S is the internal state
    A, A', S, S' = args 
    #customize your own loss components
    loss1 = K.mean(K.square(A - A'), axis=-1)
    loss2 = K.mean(K.square(S - S'), axis=-1)
    #adjust the weight between loss components
    return 0.5 * loss1 + 0.5 * loss2

 def model():
     #define other inputs
     A = Input(...) # define input A
     #construct your model 
     cnn_model = Sequential()
     ...
     # get true internal state
     S = cnn_model(prev_layer_output0)
     # get predicted internal state output
     S' = Dense(...)(prev_layer_output1)
     # get predicted A output
     A' = Dense(...)(prev_layer_output2)
     # customized loss function
     loss_out = Lambda(customized_loss, output_shape=(1,), name='joint_loss')([A, A', S, S'])
     model = Model(input=[...], output=[loss_out])
     return model

  def train():
      m = model()
      opt = 'adam'
      model.compile(loss={'joint_loss': lambda y_true, y_pred:y_pred}, optimizer = opt)
      # train the model 
      ....