From webcam to Flask, one frame a second

A trained model in a notebook is a file, not an app. To read emotions live, a frame had to travel from a webcam in the browser to a Keras model in Python, and arrive looking like the faces the model learned from.

The loop

1. The live page asks for the front camera and shows it in a 320 × 240 video box.
2. Every 1,000 milliseconds, it copies the current frame onto a hidden canvas and saves it as a JPEG “data URL”: the image written out as text.
3. It posts that text to a Flask route, /predict, as JSON.
4. The server strips the data-URL header, decodes the text back into bytes, and OpenCV turns the bytes into an image.
5. The model predicts, and the server replies with one word.
6. The page shows “Prediction: happy” under the video.

No streaming server and no websockets: a timer, a fetch call and a small JSON reply.

Same preprocessing, every time

The model was trained on 48 × 48 greyscale images, with pixels scaled from 0 to 1. A live frame is colour, 320 × 240, with pixels from 0 to 255. One small function closes that gap:

grey  = convert frame to greyscale
small = resize grey to 48 × 48
x     = small / 255, shaped (1, 48, 48, 1)

That last shape means one image with one grey channel, exactly what the model was built for. The photo-upload route and the live route both call this same function, so there is one definition of what the model sees.

The label-order trap

The model doesn’t answer “happy”. It answers with the position of its highest score, a number from 0 to 6. In the notebook, a label encoder turned folder names into those numbers, and it sorts them alphabetically: angry 0, disgust 1, fear 2, happy 3, neutral 4, sad 5, surprise 6.

So the app’s lookup table has to follow that exact order, and it does. Swap two entries and a sad face comes back as neutral, with no error anywhere. Nothing crashes; it’s just quietly wrong, which is why it’s worth checking first.

What still doesn’t match

The biggest gap is framing. FER-2013 images are faces. A webcam frame is a face plus a room, and my live route resizes the whole frame straight to 48 × 48, background and all.

The shape is off too. The frame is 4:3 and the model’s input is square, so the resize squeezes everything to three quarters of its width. Faces arrive narrower than the ones the model learned from.

The fix is first on my list: find the face, crop to it, then resize.

Smaller choices

• Uploads are capped at 5 MB, filenames are sanitised, and on phones the file picker can open the camera directly.
• The live endpoint fails clearly. A missing image or one that won’t decode gets a 400 error with the reason.
• Live frames stay in memory. The live route never writes a frame to disk. The upload route does save each photo, which I tested with 23 faces from FER-2013’s test folders, covering all seven emotions. For a face app, deleting uploads after prediction is the first thing I’d change.
• The timer doesn’t wait. A fixed one-second interval fires whether or not the last answer came back. Sending the next frame only after a reply arrives would be safer if predictions ever took longer than a second.
• The model ships as a 51 MB pickle, and Flask runs in debug mode. Both are fine for a demo on my own machine and wrong for anything public. Keras’s native save format is next.

Rules I took from it

1. Write down exactly what the model saw in training, and make every route produce the same thing.
2. Label maps follow the encoder, not memory.
3. Match the framing, not just the size. A resized room is not a face.
4. Polling once a second is a fine first live mode. Streaming can wait until it’s needed.