When more epochs stopped helping

I trained my emotion CNN for 200 epochs, and for more than half of them it was getting better at the wrong thing. Training accuracy kept climbing. The score that mattered had stopped moving long before.

The run

• 28,709 training faces in batches of 128: 225 updates per epoch.
• 200 epochs, the Adam optimiser, and dropout after every block.
• After each epoch, the model was scored on the 7,178 test faces, passed in as validation data. I’ll come back to that.

When it finished, I plotted accuracy and loss for both. The figures below are read off those charts, so treat them as approximate.

What the curves showed

• Epochs 0 to 34: both curves rise together, and validation is slightly ahead. That’s normal with dropout, which handicaps the network while it trains but not while it’s scored.
• Around epoch 34: training accuracy overtakes validation and never looks back.
• Epochs 50 to 200: training accuracy gains about 18 points, to ~82%. Validation gains about three and flattens in the low 60s.
• Loss tells it more sharply. Validation loss bottoms out at about 1.0 around epoch 85, then drifts back up to about 1.08, while training loss keeps falling to 0.53.

That’s the textbook picture of overfitting: past a point, the network stops learning about faces and starts memorising the training set.

Loss and accuracy disagreed

After epoch 85, validation accuracy still crept up by about a point while validation loss rose. Both can be true at once. Accuracy only asks whether the top guess was right; loss also asks how much probability the model put on the right answer. Rising loss with flat accuracy usually means the model is getting more confident everywhere, including on the faces it gets wrong. For an app that shows its answer to a person, that’s the wrong direction.

The rule I wrote down and didn’t follow

My presentation notes from the time say it plainly: if validation loss rises while training loss falls, the model is memorising rather than generalising, and that’s the moment to stop early. The same notes describe my loss graph as healthy convergence. Looking at it again, it isn’t. From epoch 85 to 200, 115 epochs, more than half the run, went into memorising.

A later rerun in the same notebook was set to 600 epochs. At the logged speed of about 0.74 seconds a step, that’s more than 27 hours of training, and it was stopped during the first epoch. The 200-epoch curves had already answered the question: more epochs would only have cost time.

The test set was also the validation set

The notebook passes the test split in as validation data. That’s convenient, but anything decided by watching those curves, such as when to stop, ends up tuned to the very faces the final score comes from. So about 62% is a validation number, not a clean test score.

The fix

hold out 10% of the training faces for validation
train with early stopping:
    watch validation loss
    stop after 10 epochs with no improvement
    restore the best weights
score once on the untouched test set

On this run, that would have stopped at around epoch 95 and kept the weights from around epoch 85: less than half the training, for a validation accuracy within about a point of where it finished.

Rules I took from it

1. Watch validation loss, not training accuracy. Training accuracy only tells you the model can memorise.
2. Rising validation loss with flat accuracy means overconfidence. Stop there.
3. Keep validation and test separate. One is for decisions, the other is for the score.
4. More epochs are a cost, not a fix.