vox | khushi keswani

what i built

turn detection — silero vad on 512-sample chunks (32 ms at 16 khz). a turn ends after 10 consecutive silent chunks, about 320 ms.
overlapped response — gemini streams tokens, and each finished sentence is pushed to a queue that a piper worker thread is already speaking from.
instrumented end to end — vad compute, whisper time, time to first llm chunk, time to first audio and total turn latency are all printed.
measured trade-offs — whisper sizes, faster-whisper int8, thread counts and an alternative llm provider were each tested, and several were rejected.

how it fits together

figure 1capture, detection and playback are separate producers and consumers. speech starts playing while the llm is still generating.(scroll the figure sideways)

keep the audio callback empty

the audio driver calls back every 32 ms and expects an immediate return. any real work there causes dropped frames. the callback only copies and enqueues; vad and everything else happen on the consumer side.

def callback(indata, frames, time, status):
    if status:
        print(status)
    audio = indata[:, 0].copy()
    audio_queue.put(audio)

voice_pipeline.py

vad fires late, so keep a pre-buffer

vad can only say "speech" after it has heard some, so the first syllable is already gone. a deque(maxlen=2) keeps the most recent chunks, and when speech is detected they're prepended to the utterance. without it, the transcript loses the start of words.

pre_buffer = deque(maxlen=2)
...
if speech_probability > 0.5 and not speaking:
    speaking = True
    utterance_chunks.extend(list(pre_buffer))

voice_pipeline.py

start speaking before the answer is finished

the llm is capped at 25 output tokens with minimal thinking, and the system prompt asks for one or two spoken-style sentences. as tokens stream in, text is buffered until it ends in ., ? or !, then handed to piper. the user hears sentence one while sentence two is still being generated.

sentence_buffer += chunk.text
if sentence_buffer.endswith((".", "?", "!")):
    tts_queue.put(sentence_buffer)
    sentence_buffer = ""

voice_pipeline.py

what the experiments said

tried	outcome
whisper small vs base	moved to base. load and transcribe cost outweighed the accuracy gain, and the llm absorbs most mishearings.
faster-whisper + int8	rejected for worse transcription quality.
unconstrained language	whisper produced wrong-language output; forcing `language="en"` fixed it.
openrouter instead of gemini	about 39 s observed latency and no current-information support. rejected.
quantizing the slowest layer	profiled first, quantized the hot layer, and got a worse result.

from learnings.md in the repo.

the last row is the useful one: the fastest model is not the one with fewer parameters.

metrics

stage	runs at
audio chunk	512 samples · 32 ms at 16 khz
vad	silero, speech probability above 0.5
pre-buffer	`deque(maxlen=2)` — 64 ms kept before the trigger
end of turn	10 consecutive silent chunks · about 320 ms
stt	whisper base, language forced to `en`
llm	streamed, capped at 25 output tokens
tts	piper, one sentence at a time on a worker thread

the constants the pipeline runs on. each stage is timed per turn — vad compute, whisper time, time to first llm chunk, time to first audio and total turn latency.

one number was measured and rejected outright: the openrouter path answered in about 39 s, against a turn budget of a few hundred milliseconds. that route was dropped rather than tuned.

the interesting problem was never accuracy — it was latency, in every stage.

next project → mamba, from scratch all work ↑