Measuring Latency#

This page defines the latency terms used across the kiosk UI, the GET /api/v1/pipeline/latest API, and make benchmark, and explains how to read a benchmark run.

Use these terms as written. Do not introduce synonyms — the same span having two names in two places is what made earlier numbers incomparable.

Terms#

Term	Definition
Voice-to-voice latency	Customer’s last word → first sound ready for the speaker
Endpointing delay	Customer’s last word → turn-end decision
Processing latency	Turn-end decision → first sound ready for the speaker
Transcription latency	Last word → final transcript available
LLM TTFT	Time to first token
TTS time to first sentence	Time until sentence 1’s synthesized audio WAV is on disk (not the first audio byte)

The two parts reconcile exactly:

voice_to_voice_ms == endpointing_delay_ms + processing_latency_ms

The clock stops when the first answer audio is ready for the speaker — the WAV is on disk and queued for playback. make benchmark is headless, so this is the furthest point it can observe. In the live hands-free UI the browser additionally reports when playback actually began, recorded separately as voice_to_voice_speaker_ms.

“Time to first audio” is retired. It named different spans in different places. Use voice-to-voice latency, and state which of its two parts you mean.

The opener#

By default the kiosk plays a short cached acknowledgement (“One moment.”) as soon as it decides the turn has ended, before the answer is ready. When it does, the first sound ready for the speaker is the opener, so voice_to_voice_ms measures the opener — which is a real thing the customer hears, but it is not a measure of the pipeline.

Every turn therefore reports both, and says which it was:

Field	Meaning
`voice_to_voice_ms`	To the first sound ready for the speaker, whatever it was
`voice_to_voice_answer_ms`	To the first sound of the answer
`first_audio_was_opener`	Whether the first sound was the opener
`opener_failed`	Opener was enabled but could not be rendered, so the customer heard nothing until the answer

Quote voice_to_voice_answer_ms when comparing pipeline changes. Quote voice_to_voice_ms when describing what the customer experiences.

Disable the opener with KIOSK_CORE_OPENER_ENABLED=false to measure the pipeline alone.

Host prerequisites#

Everything in this stack runs in Docker except the measurement harness, which runs on the host: the orchestrator re-invokes the in-application harness with sys.executable, so the host interpreter needs its own dependencies.

make benchmark handles this automatically. It builds a small venv (.venv-benchmark, from tests/benchmarks/requirements.txt) and launches the orchestrator with it, so the run no longer depends on what the system python3 happens to have installed. Two things are checked before the stack is started, by make check-benchmark-env:

Requirement	Why	If missing
`websockets`	Default `--uplink ws` (the browser’s hands-free path)	Installed into `.venv-benchmark` automatically
`ffmpeg`	Resamples synthesised prompts to 16 kHz mono PCM, and locates the true end of speech via `silencedetect`	Install it yourself — not a PyPI package: `make install-host-deps`

ffmpeg is not optional plumbing. Besides resampling, true_end_of_speech_seconds() runs silencedetect to find where the speaker actually stopped, from the waveform — the zero point for voice-to-voice latency, deliberately independent of the pipeline’s own VAD. Without it that function returns None and the harness silently falls back to detector-based timing, i.e. the pipeline grades its own endpointing and the numbers stop being comparable with other runs.

Both are verified up front deliberately. A missing host dependency is not a property of the pipeline being measured, and discovering it mid-run used to cost a full stack bring-up and a discarded warmup before surfacing as a report in which every turn carried the same error — which reads like a broken stack rather than a broken host.

Useful targets:

make check-benchmark-env   # verify the host, start nothing
make install-host-deps     # apt-get ffmpeg + python3-venv (uses sudo)
make benchmark-venv        # (re)build .venv-benchmark

install-host-deps is opt-in and is never a prerequisite of another target: nothing else in this repo installs system packages, so escalating to sudo stays something you type rather than a side effect of make benchmark.

Service preconditions#

Once the stack is up, the harness also requires text-to-speech and audio-analyzer to be healthy before the first turn. The orchestrator’s own gate only covers kiosk-core, rag-service and ovms-llm, so a crash-looping TTS container still prints All services healthy — and then every turn fails identically on its first synth call. That check now raises an environment error instead, naming the service and its health URL.

On a host with no PyPI access, V2V_UPLINK=rest make benchmark replays over the legacy chunked-WAV transport, which is pure stdlib. It still needs ffmpeg, and it measures a different uplink — never mix its numbers with ws-uplink numbers in one comparison.

Reading a benchmark run#

make benchmark writes per-run artifacts to results/:

File	Contents
`smart_kiosk_v2v_summary_<ts>.csv`	Per-turn latency rows
`smart_kiosk_v2v_results_<ts>.json`	Full per-turn traces and percentiles
`run_manifest.json`	What actually ran, and whether the run completed
`measured_window.json`	First and last measured turn
`consolidated_metrics.csv`	Hardware counters plus aggregate latency

Report the median and p95, never the mean — the distribution has a long tail and the mean hides it.

The headline voice-to-voice number the benchmark prints is clip-anchored: the customer’s last word is taken from the test clip itself (where speech ends in the WAV), not from the kiosk’s own voice activity detector. Endpointing is part of what is being measured, so the clip anchor is the honest customer-facing figure. The detector-anchored number (voice_to_voice_ms, started at the kiosk’s VAD) is printed below it only as a secondary cross-check. Live-mic runs have no clip ground truth and fall back to the detector number as the headline.

`run_manifest.json`#

Records what the run actually used, not what was requested. Where the orchestrator overrides a setting, both the effective value and a *_requested value appear. status is "started" until the run completes successfully, so a failed run cannot be mistaken for a good one on the strength of leftover numbers from the previous run.

`early_commit_count` — check this first#

In a benchmark run the customer’s last word is taken from the test clip itself (where speech ends in the WAV), not from the kiosk’s own voice activity detector. Endpointing is part of what is being measured, so measuring it against its own decision would always report roughly zero.

Because that anchor is independent of the kiosk, the endpointing delay can come out negative. That means the turn was committed before the customer finished speaking — the transcript is truncated and the order was taken from a partial sentence.

Field	Meaning
`early_commit_count`	Turns committed before the customer finished
`early_commit_eligible_count`	Turns with a usable clip anchor

early_commit_count must be zero. A non-zero value is a failed run no matter how good the latency percentiles look — cutting customers off shortens every measured turn, so the numbers improve as the behaviour gets worse.

Two settings keep it at zero:

KIOSK_CORE_ENDPOINT_STABLE_SECONDS — the turn is only committed once the transcript has stopped changing. While more speech is arriving the transcript keeps changing and the window resets, which is what distinguishes a mid-sentence pause from the end of a turn. A fixed silence threshold cannot tell those apart.
KIOSK_CORE_ADAPTIVE_FLUSH_PAUSE_SECONDS — must not be set below 0.50; shorter pauses are normal mid-sentence.

Run size#

A p95 over four turns is just the slowest of the four. The default benchmark runs order-fastpath: six single-intent ordering turns (one clean mutation each, ending in a confirm), all synthesised with one TTS voice, repeated V2V_RUNS times — 48 measured turns at the default V2V_RUNS=8, enough for a stable median and a p95 that is not a single outlier. This is the clean-ordering voice-to-voice number; quote it with the scenario named next to it (order-fastpath), because it deliberately stays on the fast path and is faster than a mixed workload.

For the broader workload — single items, multi-item orders, changes, removals, confirmations, mid-sentence pauses and a price-question + upsell journey — run make benchmark V2V_SCRIPT=all (88 measured turns per run) and report that number under its own name (all). Warm-up turns are pruned before the percentiles are computed in either case.

Hardware counters#

Counters are trimmed to the measured window before consolidation, so utilization reflects the turns rather than being diluted by the health wait, model warm-up and teardown.

kiosk-core runs as uid 1000. If results/ is not writable by that uid, latency events cannot be recorded and the benchmark reports zero transactions even though every turn succeeded. A warning is logged at startup if this is the case.