Build a Local Voice Assistant

What you will build

A push-to-record local voice pipeline: capture a short WAV, transcribe it with faster-whisper, ask Ollama, synthesize a response with Piper, and play it. Keeping each stage explicit makes failures easier to find than hiding everything inside one always-listening application.

This guide does not add a wake word, streaming conversation, or speaker recognition. Allow 2-3 hours plus downloads. Models are downloaded from external sources; after setup, the chosen inference path can operate locally.

Before you start

Use Linux with a working microphone and speakers, FFmpeg, Python 3.11+, and a loopback Ollama model such as llama3.2:3b. Sixteen GB RAM is a reasonable starting point. The CPU path is slower but avoids CUDA setup for the first run.

Create a fresh voice-lab directory and virtual environment:

python3 -m venv .venv
.venv/bin/python -m pip install faster-whisper piper-tts
.venv/bin/python -m pip freeze > requirements.lock.txt
ffmpeg -version
ollama list

Install FFmpeg through your OS's supported packages if absent. Confirm the Piper command with .venv/bin/piper --help; interface changes should be resolved against the installed package's documentation, not by guessing flags.

1. Record and listen first

On an ALSA-compatible Linux desktop:

ffmpeg -f alsa -i default -t 8 -ar 16000 -ac 1 question.wav
ffplay -nodisp -autoexit question.wav

Choose the correct input device if the desktop does not expose an ALSA default. Headless servers may have no audio device at all. Do not continue until you can hear the recorded question clearly, without clipping or long silence.

Do not record other people without the required permission. These WAV files can contain sensitive personal information.

2. Transcribe the recording

Download voice_step.py into the lab directory. It transcribes with CPU/int8 faster-whisper and sends text to a fixed loopback Ollama endpoint. Inspect it, then run:

.venv/bin/python voice_step.py question.wav

The first run downloads the ASR model. The program writes transcript.txt and answer.txt and prints stage timings. Listen again and compare the transcript with what you actually said.

Checkpoint: check names, numbers, and negation. If the transcript is wrong, changing the chat model is not the right first fix.

3. Install a voice you are allowed to use

Obtain a Piper voice model and its matching JSON configuration from the voice project's official model repository. Review the model card and license; code licensing does not automatically grant identical rights to every voice.

For an English example, download en_US-lessac-medium.onnx and en_US-lessac-medium.onnx.json from the Piper voices repository. Keep them together under a local voices directory and record their revision or hashes.

.venv/bin/piper --model voices/en_US-lessac-medium.onnx \
  --output_file answer.wav < answer.txt
ffplay -nodisp -autoexit answer.wav

Verify the command's flags with your installed version. Never substitute an unreviewed shell command supplied by a model response.

4. Measure the experience

Record capture duration, ASR load and transcription time, model-response time, TTS time, and playback start. Separate cold model downloads and loading from warm inference.

Repeat with a short factual request, a longer request, and an intentionally noisy recording. Keep response lengths short so speech output remains usable. A faster ASR engine does not remove recording, network, generation, or synthesis delays.

When it breaks

Empty or poor audio points to the capture device or levels. ASR memory errors call for a smaller model or more headroom. Ollama connection failures belong to the chat stage. A Piper model-load failure often means a missing or mismatched configuration file.

Feedback loops occur when the microphone records the assistant's own speaker output. Use headphones or separate recording and playback. Do not add an always-listening loop until cancellation, consent, and echo handling are understood.

Cleanup and next steps

No persistent voice service is installed. Keep or delete the disposable audio and text files deliberately, and protect model caches according to your retention policy. Unload the chat model if needed.

For document-grounded voice answers, connect the transcription stage to the RAG guide, preserving source references separately from spoken summaries.

Explore theLAB