How to tell which Keras model is better?

Question

I don't understand which accuracy in the output to use to compare my 2 Keras models to see which one is better.

Do I use the "acc" (from the training data?) one or the "val acc" (from the validation data?) one?

There are different accs and val accs for each epoch. How do I know the acc or val acc for my model as a whole? Do I average all of the epochs accs or val accs to find the acc or val acc of the model as a whole?

Model 1 Output

Train on 970 samples, validate on 243 samples Epoch 1/20 0s - loss: 0.1708 - acc: 0.7990 - val_loss: 0.2143 - val_acc: 0.7325 Epoch 2/20 0s - loss: 0.1633 - acc: 0.8021 - val_loss: 0.2295 - val_acc: 0.7325 Epoch 3/20 0s - loss: 0.1657 - acc: 0.7938 - val_loss: 0.2243 - val_acc: 0.7737 Epoch 4/20 0s - loss: 0.1847 - acc: 0.7969 - val_loss: 0.2253 - val_acc: 0.7490 Epoch 5/20 0s - loss: 0.1771 - acc: 0.8062 - val_loss: 0.2402 - val_acc: 0.7407 Epoch 6/20 0s - loss: 0.1789 - acc: 0.8021 - val_loss: 0.2431 - val_acc: 0.7407 Epoch 7/20 0s - loss: 0.1789 - acc: 0.8031 - val_loss: 0.2227 - val_acc: 0.7778 Epoch 8/20 0s - loss: 0.1810 - acc: 0.8010 - val_loss: 0.2438 - val_acc: 0.7449 Epoch 9/20 0s - loss: 0.1711 - acc: 0.8134 - val_loss: 0.2365 - val_acc: 0.7490 Epoch 10/20 0s - loss: 0.1852 - acc: 0.7959 - val_loss: 0.2423 - val_acc: 0.7449 Epoch 11/20 0s - loss: 0.1889 - acc: 0.7866 - val_loss: 0.2523 - val_acc: 0.7366 Epoch 12/20 0s - loss: 0.1838 - acc: 0.8021 - val_loss: 0.2563 - val_acc: 0.7407 Epoch 13/20 0s - loss: 0.1835 - acc: 0.8041 - val_loss: 0.2560 - val_acc: 0.7325 Epoch 14/20 0s - loss: 0.1868 - acc: 0.8031 - val_loss: 0.2573 - val_acc: 0.7407 Epoch 15/20 0s - loss: 0.1829 - acc: 0.8072 - val_loss: 0.2581 - val_acc: 0.7407 Epoch 16/20 0s - loss: 0.1878 - acc: 0.8062 - val_loss: 0.2589 - val_acc: 0.7407 Epoch 17/20 0s - loss: 0.1833 - acc: 0.8072 - val_loss: 0.2613 - val_acc: 0.7366 Epoch 18/20 0s - loss: 0.1837 - acc: 0.8113 - val_loss: 0.2605 - val_acc: 0.7325 Epoch 19/20 0s - loss: 0.1906 - acc: 0.8010 - val_loss: 0.2555 - val_acc: 0.7407 Epoch 20/20 0s - loss: 0.1884 - acc: 0.8062 - val_loss: 0.2542 - val_acc: 0.7449 

Model 2 Output

Train on 970 samples, validate on 243 samples Epoch 1/20 0s - loss: 0.1735 - acc: 0.7876 - val_loss: 0.2386 - val_acc: 0.6667 Epoch 2/20 0s - loss: 0.1733 - acc: 0.7825 - val_loss: 0.1894 - val_acc: 0.7449 Epoch 3/20 0s - loss: 0.1781 - acc: 0.7856 - val_loss: 0.2028 - val_acc: 0.7407 Epoch 4/20 0s - loss: 0.1717 - acc: 0.8021 - val_loss: 0.2545 - val_acc: 0.7119 Epoch 5/20 0s - loss: 0.1757 - acc: 0.8052 - val_loss: 0.2252 - val_acc: 0.7202 Epoch 6/20 0s - loss: 0.1776 - acc: 0.8093 - val_loss: 0.2449 - val_acc: 0.7490 Epoch 7/20 0s - loss: 0.1833 - acc: 0.7897 - val_loss: 0.2272 - val_acc: 0.7572 Epoch 8/20 0s - loss: 0.1827 - acc: 0.7928 - val_loss: 0.2376 - val_acc: 0.7531 Epoch 9/20 0s - loss: 0.1795 - acc: 0.8062 - val_loss: 0.2445 - val_acc: 0.7490 Epoch 10/20 0s - loss: 0.1746 - acc: 0.8103 - val_loss: 0.2491 - val_acc: 0.7449 Epoch 11/20 0s - loss: 0.1831 - acc: 0.8082 - val_loss: 0.2477 - val_acc: 0.7449 Epoch 12/20 0s - loss: 0.1831 - acc: 0.8113 - val_loss: 0.2496 - val_acc: 0.7490 Epoch 13/20 0s - loss: 0.1920 - acc: 0.8000 - val_loss: 0.2459 - val_acc: 0.7449 Epoch 14/20 0s - loss: 0.1945 - acc: 0.7928 - val_loss: 0.2446 - val_acc: 0.7490 Epoch 15/20 0s - loss: 0.1852 - acc: 0.7990 - val_loss: 0.2459 - val_acc: 0.7449 Epoch 16/20 0s - loss: 0.1800 - acc: 0.8062 - val_loss: 0.2495 - val_acc: 0.7449 Epoch 17/20 0s - loss: 0.1891 - acc: 0.8000 - val_loss: 0.2469 - val_acc: 0.7449 Epoch 18/20 0s - loss: 0.1891 - acc: 0.8041 - val_loss: 0.2467 - val_acc: 0.7531 Epoch 19/20 0s - loss: 0.1853 - acc: 0.8072 - val_loss: 0.2511 - val_acc: 0.7449 Epoch 20/20 0s - loss: 0.1905 - acc: 0.8062 - val_loss: 0.2460 - val_acc: 0.7531 

Answer 1

Do I use the "acc" (from the training data?) one or the "val acc" (from the validation data?) one?

If you want to estimate the ability of your model to generalize to new data (which is probably what you want to do), then you look at the validation accuracy, because the validation split contains only data that the model never sees during the training and therefor cannot just memorize.

If your training data accuracy ("acc") keeps improving while your validation data accuracy ("val_acc") gets worse, you are likely in an overfitting situation, i.e. your model starts to basically just memorize the data.

There are different accs and val accs for each epoch. How do I know the acc or val acc for my model as a whole? Do I average all of the epochs accs or val accs to find the acc or val acc of the model as a whole?

Each epoch is a training run over all of your data. During that run the parameters of your model are adjusted according to your loss function. The result is a set of parameters which have a certain ability to generalize to new data. That ability is reflected by the validation accuracy. So think of every epoch as its own model, which can get better or worse if it is trained for another epoch. Whether it got better or worse is judged by the change in validation accuracy (better = validation accuracy increased). Therefore pick the model of the epoch with the highest validation accuracy. Don't average the accuracies over different epochs, that wouldn't make much sense. You can use the Keras callback ModelCheckpoint to automatically save the model with the highest validation accuracy (see callbacks documentation).

The highest accuracy in model 1 is 0.7737 and the highest one in model 2 is 0.7572. Therefore you should view model 1 (at epoch 3) as better. Though it is possible that the 0.7737 was just a random outlier.