A family dashboard that listens in Swedish, and the evals that keep it honest

How the assistant works

Safari turns speech into text on the iPad. The server sends that one sentence, today's date and the names of the family's lists and calendars to Claude, which answers with tool calls: add these items to this list, add this event, log this shared expense, show tomorrow's weather. The server checks every call before running it with the same code the dashboard's own buttons use. List contents, events and money never leave the house.

Why it needed evals

A voice assistant that's right most of the time still puts a dentist appointment on the wrong Friday. I wanted to know how often it was wrong, and in which ways, before and after every change to the prompt. So the assistant has an eval suite: 29 spoken commands with known right answers, from shopping lists and multi-day events to expenses, weather questions, undo and commands that are too vague to act on.

Each case is scored with hard rules on what matters: the right list or calendar, the exact start and end time, the amount to the öre, and a clarifying question when the payer is missing. Free text such as item wording or a recipe is checked with keywords and limits, never word for word. Every case runs three times, because the model isn't deterministic; a case that passes some runs and fails others is flagged as flaky. The suite runs in GitHub Actions on every prompt change and every Monday.

What it found

The assistant asked too much. Given "middag med farmor på lördag" it asked for a time instead of creating an all-day event, and it asked what "möte nästa måndag klockan 8" should be called. It also answered some questions in plain English text instead of the card the iPad expects. Three sentences in the prompt fixed those: act on sensible defaults, read "fredag 22 till 01" as a start and an end time, and never reply in plain text.

Two failures survived the fix, and they are the interesting ones. "Fest på fredag 22 till 01" still sometimes lands a week late, and a Saturday weather question gets tomorrow's forecast. Both are calendar arithmetic, and language models are poor at it. The next fix moves that work into code: the assistant gets a table of the next two weeks' dates instead of computing them.

What I took from it

Evals turn "it feels better" into a number you can argue with, including when the number is smaller than you hoped: six runs on 29 cases can't separate 92% from 95%. They also tell you which problems a better prompt can fix and which ones belong in code. That second answer is the more useful one.