48 test commands, each with a reason to exist

A command engine that handles the five sentences you tried is a demo. The useful question is what it does with the awkward ones: a chained command, a pronoun, a toaster, an unlock tucked into the end of a sentence.

So the engine I built for AHA’s hiring process comes with a test set of 48 commands, and most of them are awkward on purpose.

Every case says why it’s there

Each case is a small record: the command, the intent it should produce, the safety outcome it should get, and a one-line reason.

{
  "command": "Before I leave, turn off the lights and unlock the door",
  "expected_intent": "device_control",
  "expected_safety": "confirmation_required",
  "description": "Mixed safe + risky in natural language"
}

The reasons read like a checklist of ways to fail: “Pronoun resolution test”, “Vague quantitative modifier”, “Ambiguous door reference”, “Alias not defined in schema”, “Urgent risky phrasing”.

What’s in the 48

• 34 should become device control, from “Turn on the kitchen light” to “Turn on the bedroom fan and kitchen light”.
• 8 should make the engine ask instead of act: “Turn it off”, “Lower the AC slightly”, “Turn the lights blue”, “I’m going to sleep”.
• 3 are scene macros (movie, hosting and game time), 2 are things it can’t do (a coffee, a toaster) and 1 is a status question: “Are the doors locked?”
• 7 must come back flagged for confirmation, including “Disable the alarm”, “Disable security immediately” and an unlock tucked at the end of a chained command.

That’s 17 of 48 cases where the right answer is a question, a refusal or a confirmation, not an action. A test set of happy paths would have none of them.

What the harness reports

The evaluation script runs every command through the real parser, rules first and then the local Llama 3.2, and prints a row per case: the command, pass or fail, which layer answered and the confidence. The summary adds:

• Strict accuracy: did the intent match?
• Rule and LLM share: how many of the 48 the rules answered without the model, and how many needed it.
• Average confidence across the set.
• “Semantic improvements”: cases where the test expected a question but the engine resolved a real action.

That last count matters, because a strict score calls those failures and some of them aren’t. Case 16, “Turn off the garage light”, is labelled “Invalid location” and expects a question. The final schema has a garage, so the engine simply does it.

Alongside the set, 8 unit tests pin down the basics: four for the rules (a plain command, an alias, “turn off everything”, a scene trigger) and four for the guardrail (allow a light, reject a nuclear reactor, flag an unlock, allow a status check).

What it doesn’t score yet

Every case records its expected safety outcome, but the harness scores intent. So “Disable the alarm” can pass on intent alone without its confirmation ever being checked, and the guardrail’s unit tests cover an unlock, not every phrasing of one. Scoring the safety column in the same run is the obvious next step, because it’s the column that matters most.

Tests also shape the code. “Set the AC to 24 degrees” is labelled “Missing location” but expected to act, and the rules learned to default lights, AC, fans and curtains to the whole house to match. That’s a reasonable choice, as long as it’s made on purpose and not just to turn a row green.

Rules for a test set like this

1. Give every case a reason. A failing row should explain itself.
2. Test the refusals as hard as the actions. Here, that’s 17 of 48 cases.
3. Record which layer answered. A pass from the rules and a pass from the model aren’t the same pass.
4. Count “acted where it should have asked” separately. Some are wins and some are risks, so read each one.
5. Score what matters most. For a house, that’s safety, not intent.