Local Voice Assistant for Home Assistant

Overview

At first my Home Assistant satellite, the Voice Preview Edition, only played notification chimes. It became a voice assistant that understands German and English, listens for its own wake word and does all speech processing on my own GPUs. The brief was a custom wake word, no cloud and a calm, natural voice. An answer should arrive within two to five seconds, and at about 3.4 seconds it does.

I led the project. I set the goals, flashed and soldered the satellites, configured Home Assistant and chose every voice by ear. I used Claude to move faster, mainly for research, the container stack, the patches and the evaluations.

System

Satellites

Besides the Voice Preview Edition, three self-built ReSpeaker Lite boards are in use, all running ESPHome with a wake word detected directly on the ESP32-S3. On the ReSpeaker boards the mute line was not connected from the factory. Only after I set the solder bridge did a muted board really stop reacting to the wake word.

Speech

Speech-to-text runs on whisper.cpp over Vulkan, so the later switch from an NVIDIA to an AMD graphics card changed nothing for it, and it recognises a command in 365 to 522 milliseconds. Kyutai Pocket TTS handles text-to-speech on the CPU. Since replies are streamed, even a long answer starts after about one second instead of more than ten, and a dedicated speller gets numbers and times right in both languages.

Home Assistant recognises frequent commands itself in about 0.02 seconds, and only what it cannot match goes to the language model. Anything that touches privacy or safety, such as muting by voice, deliberately runs through fixed commands and never through the language model.

Knowledge service

A small Python service answers questions like “Which supermarket nearby is open right now?” from a local OpenStreetMap index and takes regional public holidays into account. It is rebuilt monthly and covered by 87 unit tests.

Stumbling blocks

Whisper invented whole sentences on near-silent audio and sometimes transcribed the assistant’s own reply, until filters for voice activity, language and confidence put an end to it. After the graphics card swap the GPU services quietly ran on the CPU, because the default runtime under rootless Podman blocked access to the card. Switching the runtime to crun fixed it.

Voice clone

Training the wake word needed many recordings of exactly the voice it will hear most, which is my own. So I had a zero-shot clone of my voice built in German and English. It started from 18 reading passages I recorded, plus a podcast and an audiobook reading. In the podcast, speaker diarisation with pyannote found my voice automatically, which left 156 reference files with 67 minutes of speech.

Chatterbox Multilingual then produced 6,000 recordings of “Cassandra” and 11,354 read-aloud book sentences without the wake word, which served as the hardest negative examples. Automatic checks sorted out doubled words, clipped endings and background sounds carried over from the references, and a blind check of 100 random clips turned up no defect at all. A by-product is a tool that reads any text aloud in my voice. The assistant itself still speaks with a stock voice.

Wake word “Cassandra”

The wake word is a model of about 138 kilobytes that runs directly on the satellites. Four generations came together in about 20 days. I had the fourth one retrained from scratch after the assistant woke up during phone calls, with the goal of matching the reliability of the stock wake word “Okay Nabu”.

Effort

The rules for the evaluation were fixed before a single result existed. The design grew to about 5,300 lines over 24 revision rounds with two independent adversarial reviewers. To measure recall I recorded twelve sittings on seven consecutive days, two a day on five of them. The sittings rotated through four rooms, and a laptop and three phones recorded every utterance at once, with the phones swapping positions on a fixed schedule.

Every sentence was spoken once with “Cassandra” and once with “Okay Nabu”, so both wake words got the same recording conditions. That produced 242 sentence pairs in 2,324 recordings. I discarded two earlier sittings because the laptop’s microphone switch was off, even though every technical check had passed. Since then a signal test checks each channel on its own.

Results

The model recognises a bare “Cassandra” 99.2 percent of the time, while “Okay Nabu” manages 92.6 percent under the same conditions, and the lead is statistically significant. When the wake word sits inside a command, the comparison stays inconclusive. On false wakes the fourth generation is far better than the third and triggered three times instead of 17 on the same dinner-party recordings. It still does not reach the bar set by “Okay Nabu”. In short, the model hears its owner better than the reference does, but wakes more often when nobody is talking to it.

LLM tuning

From a small to a large model

The first build ran a 4-billion-parameter Qwen3 model through Ollama on an RTX 2060 with 6 GB, and an answer took 169 seconds. After switching to a model without thinking mode, moving it fully onto the GPU, reducing the context to 4,096 tokens and having Home Assistant pass on only the devices it needed, a test answer took 1.47 seconds. Since September a 27B Qwen model has been running through vLLM on an RTX 5090. With thinking mode off it answers in 2.8 seconds and handles compound commands such as “turn on all lights except in the study”.

Settings that mattered

The model had to actually control Home Assistant, not just claim it had switched the light on. At the default temperature of 1.0 it called a tool in only 54 percent of cases, and at 0.3 or lower in 132 of 132. So the temperature for Home Assistant is fixed at 0.25. Every change was measured by capturing real Home Assistant requests and replaying them repeatedly with a modified prompt. That is how, for example, correctly executed commands on the small model went from 1 to 12 out of 12.

Temporary workarounds

In total 14 workarounds are documented, each with its reason and what became of it. Most disappeared with the move to the large graphics card, such as the smaller context and the hidden devices. Others stayed on purpose because they belong in the right layer, for example number pronunciation in the speech output rather than in the prompt. Once, a Home Assistant update quietly reset the temperature while the interface still showed 0.25.

Status

The assistant is in daily use on four satellites with its own wake word. Still open are that the language model only answers while the workstation is on, that the wake word does not yet meet the bar on false wakes, and that its threshold has not been measured on the ReSpeaker microphones.

Stack

Home Assistant, ESPHome, microWakeWord, Wyoming, whisper.cpp, faster-whisper, Kyutai Pocket TTS, Chatterbox, pyannote, TensorFlow, Ollama, vLLM, Qwen, Podman, Docker, Debian, Python, OpenStreetMap and SQLite.