Rules first, LLM second
People don’t talk to their homes like an API. A smart-home command engine where rules answer instantly, a local LLM handles the odd phrasing, and guardrails decide what’s allowed.
The problem
Vague voice commands are slow, and unsafe, to send straight to an LLM.
The solution
Rules answer first, a local LLM handles the rest, and guardrails decide.
I built this engine as part of AHA’s hiring process, and it set the pattern I’ve used since. The problem sounds simple, “turn speech into device actions”, until you listen to how people actually talk to a house.
Three kinds of commands
- Literal: “turn on the bedroom lights”. A regular expression handles it.
- Indirect: “I’m feeling hot”, “it’s too dark”, “make it cooler”. These need meaning, not matching.
- Compositional: “everything off except the bedroom”. This is set logic over rooms and devices.
Rules first
The rule engine runs on every command. It handles aliases and typos (“fna” → fan, “blinds” → curtains), and it understands capabilities: set_temperature means the AC. When it’s confident (0.9 or higher) it answers immediately and the LLM is never called. Most real commands end here, in milliseconds.
A semantic resolver sits on top for indirect language. Comfort complaints map to a device and an action. Comparatives like “cooler” and “brighter” become relative changes. “Everything except the bedroom” becomes the complement of a set of rooms.
LLM second, inside a fence
Only what the rules can’t parse goes to Llama 3.2, running locally on Ollama. The prompt lists exactly which devices, rooms and capabilities exist, with a few worked examples. The engine extracts the first JSON object from the reply and drops anything not on the allow-list. If confidence falls below 0.55, it doesn’t guess. It asks a targeted question, like “Which room’s light?”.
Guardrails the model can’t talk around
Safety is a separate layer that applies whoever produced the action:
- “Turn off everything” never includes locks or alarms unless they’re named.
- Unlocking and disarming need typed confirmation; voice and WhatsApp can’t do them at all.
- Unknown devices are rejected. The unit test for this uses a “nuclear reactor”.
One brain, three ways in
The same parser, guardrail and device state serve a console, offline Whisper voice (woken by “Aha, listen”) and a WhatsApp bot. A live dashboard shows every decision: which layer answered, with what confidence, and what it did.
Takeaway
LLMs are best as the exception handler, not the main path. Put fast, testable rules in front, a strict schema around the model, and safety outside both.