How to evaluate a multi input/output model in keras?

Question

I've followed the description on this guide by keras to build the following model with multi-input and multi-output.

## define the model
EMBEDDING_SIZE = 128
HIDDEN_LAYER_SIZE = 64
BATCH_SIZE = 32
NUM_EPOCHS = 10

# first input model
main_input = Input(shape=(50,), dtype='int32', name='main_input')
embedding = Embedding(MAX_NB_WORDS, EMBEDDING_SIZE,
                    input_length=MAX_SEQUENCE_LENGTH)(main_input)
lstm_out = LSTM(HIDDEN_LAYER_SIZE)(embedding)
auxiliary_output = Dense(4, activation='sigmoid', name='aux_output')(lstm_out)
# second input model
auxiliary_input = Input(shape=(5,), name='aux_input')
x = concatenate([lstm_out, auxiliary_input])

x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)
x = Dense(64, activation='relu')(x)

main_output = Dense(4, activation='sigmoid', name='main_output')(x)

model = Model(inputs=[main_input, auxiliary_input], outputs=[main_output, auxiliary_output])

model.compile(optimizer='rmsprop',
              loss={'main_output': 'categorical_crossentropy', 'aux_output': 'categorical_crossentropy'},
              loss_weights={'main_output': 1., 'aux_output': 0.2})

model.fit([x1train, x2train], [ytrain, ytrain],
                    epochs=NUM_EPOCHS, batch_size=BATCH_SIZE,
                    validation_data=([x1test, x2test], [ytest, ytest]))

In the next step, I want to evaluate my model as well. I suggested running this code for it:

score, acc = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                            y={'main_output': ytest, 'aux_output': ytest},
                            batch_size=BATCH_SIZE)

But with that code I am getting the error "ValueError: too many values to unpack (expected 2)"

So I thought I maybe get a score and accuracy for both outputs and tried this code, too:

score1, score2, acc1, acc2 = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                            y={'main_output': ytest, 'aux_output': ytest},
                            batch_size=BATCH_SIZE)

but now I am getting the error "ValueError: not enough values to unpack (expected 4, got 3)"

So what am I doing wrong? I am just interested in the accuracy of my main_output to be honest.

Answer 1

From keras documentation for evaluate the can be found here

Returns

Scalar test loss (if the model has a single output and no metrics) or list of scalars (if the model has multiple outputs and/or metrics). The attribute model.metrics_names will give you the display labels for the scalar outputs.

According to your model, if you do print(model.metrics_names) you will get ['loss', 'main_output_loss', 'aux_output_loss'].

The model.evaluate produces a scalar of this format, which indicates what each of those numbers you see in the output of evaluate method corresponds to.

Hence your code,

score1, score2, acc1, acc2 = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                            y={'main_output': ytest, 'aux_output': ytest},
                            batch_size=BATCH_SIZE)

will result in an error because there are only 3 indices in the scalar and the code expects to find 4.

Also,

score, acc = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                            y={'main_output': ytest, 'aux_output': ytest},
                            batch_size=BATCH_SIZE)

will result in an error due to the fact there are more values returned by the evaluate.

You can do something like this if you want to unpack the result of evaluate in your model directly.

loss, main_loss, aux_loss = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                            y={'main_output': ytest, 'aux_output': ytest},
                            batch_size=BATCH_SIZE)

Also in your code, I can see acc1 and acc2 which makes me assume that you are expecting to evaluate the model accuracy.

Having said that, I can see that you have not used any metrics for compilation of the model. If you want to evaluate acc of the model then compile your model like this.

model.compile(optimizer='rmsprop',
              loss={'main_output': 'categorical_crossentropy', 'aux_output': 'categorical_crossentropy'},
              loss_weights={'main_output': 1., 'aux_output': 0.2}, metrics=['acc'])

Then on model.evaluate() you will get a scalar corresponding to

['loss',
 'main_output_loss',
 'aux_output_loss',
 'main_output_acc',
 'aux_output_acc']

Hence, you can unpack it like this,

loss, main_loss, aux_loss, main_acc, aux_acc = model.evaluate(x={'main_input': x1test, 'aux_input': x2test},
                                y={'main_output': ytest, 'aux_output': ytest},
                                batch_size=BATCH_SIZE)

Answer 2

You haven't defined in the model compile that you want accuracy, so when you use evaluate the function will return the loss. You have 3 losses here which evaluate returns:

• (1) Weighted averaged loss of (2) and (3), in your case: 1. * (2) + 0.2 * (3)
• (2) Loss of main_output - categorical_crossentropy
• (3) Loss of aux_output - categorical_crossentropy