Console, offline voice and WhatsApp on one command engine

A house gets talked to in different ways: typed at from a laptop, spoken to from the sofa, messaged from the office. If each channel had its own parser, the same sentence could do three different things, and a safety rule fixed in one place could stay broken in another.

One brain, passed around

In the command engine I built for AHA’s hiring process, a single controller owns three things: the parser (rules first, then a local Llama 3.2), the safety guardrail, and the simulated MQTT layer that holds device state. Every channel uses those same three objects.

• The console calls the controller directly.
• Voice calls it through a callback whenever it hears the wake phrase.
• WhatsApp runs in a small Flask web server that’s started with the same parser, guardrail and device state passed in.

Shared state is what makes it feel like one house. Switch a light on by voice, and a status question on WhatsApp reports it.

Each channel runs on its own thread: the console on the main one, the voice listener and the transcriber on two more, and the web server on another, next to a scheduler for the proactive routine.

Voice that never leaves the laptop

Whisper’s base model runs locally, on the GPU if there is one and the CPU if not. Each clip is written to a temporary file, transcribed in English and deleted.

• The microphone calibrates to the room’s noise for a second, then listens in phrases of up to 10 seconds.
• The wake phrase is “Aha, listen”. The silence detection sometimes clipped the start of a phrase, taking “Aha” with it, so the matcher also accepts “ah listen”, “aha list”, a phrase that starts with “aha” or “ah”, or “listen” on its own.
• Everything after the wake phrase becomes the command.
• Voice is off by default. Typing “voice on” in the console starts it.

WhatsApp with a menu

Twilio forwards each message to a webhook on the Flask server. “Hi”, “help” or “menu” returns a 5-item menu: living-room lights on, secure the house, movie time, a status check, and turn off everything. Sending a number runs that command; anything else is parsed as plain English.

The replies are written for a chat, not a terminal: an “Executed” card with the action, devices and rooms, a clarifying question, “I cannot do that yet” for things like coffee, and status as a sentence, such as “Currently, the Kitchen Light is ON.”

Same meaning, different permissions

The channels agree on what a sentence means. They don’t agree on what’s allowed. For the two high-risk actions, unlocking a door and disarming the alarm:

• Console: a security alert, then a typed “yes”.
• Voice: refused, because it needs console confirmation.
• WhatsApp: refused, with a security alert.

The brain decides what a command means; the channel decides whether it can happen.

A window onto all three

The same web server hosts a live dashboard. It reads one shared event log, the last 50 events, polled every second: what came in, what the engine was working on, and what it executed. Beside it sit the devices that are currently on, refreshed every two seconds, and two lists: what the engine supports, and what’s out of scope, from making coffee to ordering pizza.

What I’d keep

1. Share the brain, not the replies. WhatsApp has its own handler, because a chat needs different words from a terminal, but it calls the same parser, guardrail and state.
2. Let the channel set the permissions. Same meaning everywhere; risky actions only where someone can type a yes.
3. Forgiving wake words are fine when voice can’t confirm anything risky. A loose match costs less when anything high-risk is refused anyway.
4. Log every channel to one place. One event stream makes three channels readable as one system.