Why FER-2013 is hard, and what 62% really means
Tiny grey faces, noisy labels, and one emotion outnumbered 16 to 1. What that does to an accuracy score, and the baselines that make 62% readable.
The problem
The faces are tiny, grey, noisily labelled and badly unbalanced.
The fix
Count every class before training, and read accuracy against simple baselines.
- 7,215 vs 436
- happy vs disgust training faces
- 24.7%
- score for always answering “happy”
- ~62%
- the model’s validation accuracy
My emotion model finished at about 62% validation accuracy. On its own that sounds weak: it’s wrong almost four times in ten. But on FER-2013 the data sets the ceiling, so before defending the number or apologising for it, I needed to know what it was up against.
What a FER-2013 face is
Every image is 48 × 48 pixels in greyscale: 2,304 numbers per face. A raised eyebrow or a tightened lip is a handful of those pixels. There’s no colour, and nothing but the face.
The dataset arrives as folders, one per emotion: angry, disgust, fear, happy, neutral, sad and surprise. The folder name is the label. That gave me 28,709 training faces and 7,178 test faces, matching the counts my notebook printed as it loaded them.
The labels are also noisy. Each face gets exactly one emotion, even when the expression sits between two, so some of the “mistakes” any model makes are arguable.
Counting before training
Before building anything, I plotted how many faces each emotion had and looked at a grid of 25 labelled samples. The bar chart told the main story:
- Happy: 7,215 faces, a quarter of the training set
- Neutral, sad, fear and angry: between 3,995 and 4,965 each
- Surprise: 3,171
- Disgust: 436, just 1.5% of the training set
Happy outnumbers disgust by 16.5 to 1. The test set has the same shape: 1,774 happy faces against 111 disgust.
The baselines that make 62% readable
An accuracy figure means little without something to compare it to. Two baselines need no model at all:
- Guess at random among seven emotions: about 14.3%.
- Always answer “happy”: 24.7% on the test set, because a quarter of its faces are happy.
Against those, about 62% is roughly two and a half times what the laziest possible “model” scores. It isn’t a solved problem, but the network is clearly learning.
What overall accuracy hides
All 111 disgust test faces together are worth about 1.5 points of accuracy. A model could miss every one of them and the headline number would barely move. My notebook only measured overall accuracy, so I can’t say how well disgust was recognised, and I’d rather admit that than guess.
The imbalance shapes training too. With a plain loss function and no weighting, every face counts the same, so getting happy faces right pays off sixteen times as often as getting disgust right.
What I did, and what I’d do now
In my project presentation I listed three remedies: weight the rare classes more heavily, augment them with flipped and slightly rotated copies, or collect more balanced data. I didn’t apply them to this model. In the same internship I did use class weights on a pneumonia model with a 1 to 3 imbalance; FER-2013’s worst gap is more than five times wider.
Next time, the order would be:
- Count every class and write down the baselines first.
- Train with class weights, or augment the rare classes.
- Report recall for each emotion, not just one accuracy figure, and check which emotions get mistaken for which.
Rules I took from it
- Count classes before you train. Imbalance is cheap to see early and expensive to discover late.
- Put a baseline next to every accuracy. 62% against 24.7% says far more than 62% alone.
- Overall accuracy can’t see a 1.5% class. Per-class numbers can.
- Small, grey, noisy data has a ceiling. Judge progress against it, not against 100%.