Get Started#
This page is the entry point for running the Audio Analyzer microservice. Pick one of the two deployment paths and follow the linked guide.
Application Overview#
The Audio Analyzer microservice provides automatic speech recognition (ASR) and optional speaker diarization. It captures and transcribes audio, optionally identifying different speakers in the same audio chunk. Optional sentiment analysis per chunk complements the transcription output.
If you enable speaker diarization in config.yaml (models.asr.diarization: true), you must provide a
Hugging Face access token (HF_TOKEN) and accept the Pyannote speaker-diarization model license on
Hugging Face if you want diarization to initialize successfully. If diarization setup is incomplete,
provisioning or startup fails visibly instead of silently disabling diarization.
Before You Begin#
Confirm that your machine meets the System Requirements.
Review the Configuration Guide if you plan to change models, devices, or chunking behavior.
FunASR Paraformer with ModelScope#
FunASR is opt-in: set models.asr.provider: funasr and models.asr.name: paraformer-zh (or paraformer-en / paraformer-online) in config.yaml and
build Audio Analyzer with INSTALL_CJK=true. Mandarin uses
iic/speech_paraformer-large-vad-punc_asr_nat-zh-cn-16k-common-vocab8404-pytorch,
iic/speech_fsmn_vad_zh-cn-16k-common-pytorch, and
iic/punc_ct-transformer_zh-cn-common-vocab272727-pytorch, each pinned to the
ModelScope release revision v2.0.4. These are native PyTorch snapshots;
downloading them does not produce OpenVINO IR. Custom vad_model and
punc_model values must be ModelScope repository IDs and use their default
revision.
Where model snapshots come from#
At startup Audio Analyzer derives the snapshots it needs from config.yaml
(Whisper source for the openvino provider, a single ggml file for
whispercpp, Paraformer/VAD/punctuation for funasr, the pyannote pipeline and
its dependencies when diarization is enabled) and makes each one available in
this order:
Already on the models volume and valid — reused without any network access.
Shared Model Download service — when
AUDIO_ANALYZER_MODEL_DOWNLOAD_URLis set (for examplehttp://model-downloader:8000) and the service answersGET /health, each snapshot is requested withPOST /models/download(huggingfacefor Whisper sources and diarization,modelscopefor FunASR) and the job is polled until it completes. Both containers must mount the same storage: the service’sMODELS_DIR(default/opt/models) and Audio Analyzer’s/app/audio_analyzer/models. Enable thehuggingfaceandmodelscopeplugins on the service. Gated diarization models forwardHF_TOKENper request as anoverride_credentialsentry. The internal call bypasseshttp_proxy/https_proxy; setAUDIO_ANALYZER_MODEL_DOWNLOAD_TIMEOUT_SEC(default1800) if large downloads need longer. This is how Smart Classroom shares one Model Download container between its text-generation and ASR services.Direct download inside Audio Analyzer — the default when no service URL is set, and the fallback (with a warning) when the configured service is unreachable. Audio Analyzer calls the same Hugging Face / ModelScope
snapshot_downloadthe Model Download plugins use, with the same pinned revisions, so the result on disk is identical. This keeps deployments without a Model Download container (the standalone Compose file in this repository, VSS, Helm charts, bare-metal runs) working with no extra configuration. SetAUDIO_ANALYZER_MODEL_DIRECT_DOWNLOAD=falseto forbid in-container downloads; Audio Analyzer then fails fast if a snapshot is missing (or if the configured service is unreachable).
Snapshots live under models/<destination>/<hub>/<owner_model>/
(whisper-source/, funasr/, diarization/) with a completion inventory that
records the requested model, revision, and file sizes. Missing, truncated, or
mismatched snapshots are rebuilt by the active source; valid ones are retained.
The inventory detects incomplete files, not same-size content tampering or
upstream changes to an unpinned branch. Existing exported OpenVINO models are
reused locally. Existing Hugging Face FunASR snapshots are not migrated or
deleted: the first ModelScope download goes into models/funasr/modelscope/ and
requires additional disk space.
OpenVINO conversion runs locally in Audio Analyzer from the downloaded source
snapshot (past-aware ASR export for OpenVINO GenAI). For gated pyannote,
provide HF_TOKEN (or models.asr.hf_token); tokens are not required to load
an already provisioned diarization pipeline. Standalone diarization dependencies
are discovered from the downloaded pipeline configuration; Audio Analyzer writes
a separate local pipeline configuration and leaves the downloaded one unchanged.
Native OpenAI Whisper checkpoints and the optional sentiment model keep their
own direct-download behavior and are not affected by these settings.
After changing the selected model, companions, or diarization settings, recreate Audio Analyzer; it provisions whatever is missing on the next start:
docker compose up -d --force-recreate audio-analyzer
If provisioning fails, inspect docker compose logs audio-analyzer (and the
Model Download service logs when one is configured), correct the configuration,
credentials, or network access, and restart. Do not delete the models volume
when restarting the services.
Choose Deployment Path#
The container image exposes the API on host port 8010 and mounts shared
folders for models, chunks, storage, and the Hugging Face cache.
Fresh clones include placeholder directories for these mount roots. If you
delete them and then start Compose, Docker may recreate the missing host
paths as root before the container starts.
See Run with Docker Compose for the full step-by-step guide.
Quick start:
docker compose up -d --build
curl --noproxy '*' http://127.0.0.1:8010/health
If you hit permission errors on models/, chunks/, storage/, or
.cache/huggingface/, see
Troubleshooting.
Device visibility note: The Docker Compose flow is the verified path for GPU and NPU acceleration. The container image includes the OpenVINO GPU and NPU runtime libraries and exposes
/dev/driby default, soopenvino.Core().available_devicesinside the container reportsCPU,GPU, andNPU(when the host drivers are present).When running directly from the host
.venvwithout the full Intel GPU/NPU runtime stack installed on the host, OpenVINO may report onlyCPU. In that case, starting the service withdevice: GPUordevice: NPUfails fast with:RuntimeError: Configured OpenVINO ASR device 'GPU' is not visible in this runtime.This is a host runtime environment limitation — no application or configuration change is required. Use Docker Compose for the accelerator-enabled setup.
NPU users: before starting the container, set
ACCEL_MOUNT_PATHin your.env(or export it before running Compose) to the host NPU device node (for example/dev/accel/accel0on many Meteor Lake systems). Without it, the container will fail to start whendevice: NPUis configured. See Configuration Guide for details.
Run the service directly with Python. This path is useful for development or when you do not want to use Docker.
See Run on the Host for the full step-by-step guide.
Quick start:
python -m venv .venv
source .venv/bin/activate
pip install -r requirements.txt
python main.py
Verify#
Once the service is running:
curl --noproxy '*' http://127.0.0.1:8010/health
Expected response:
{"status": "ok"}
Verify batch transcription#
Transcribe a file and get a single JSON response:
curl --noproxy '*' -X POST http://127.0.0.1:8010/v1/audio/transcriptions \
-F file=@tests/philosophy_10_russell_128kb.mp3
Expected result: a JSON body containing a "text" field, plus an X-Session-ID
response header.
Verify OpenAI-compatible streaming (SSE)#
Add stream=true to receive incremental Server-Sent Events instead of
waiting for the whole file. Use -N so curl does not buffer the stream:
curl --noproxy '*' -N -X POST http://127.0.0.1:8010/v1/audio/transcriptions \
-F file=@tests/philosophy_10_russell_128kb.mp3 \
-F stream=true
Expected result: a sequence of transcript.text.delta frames as each chunk is
transcribed, one final transcript.text.done frame with the full text, then
the [DONE] sentinel:
data: {"type": "transcript.text.delta", "delta": "Chapter X of Philosophy by Bertrand Russell..."}
data: {"type": "transcript.text.delta", "delta": "The electron, which has been moving in one orbit..."}
data: {"type": "transcript.text.done", "text": "Chapter X of Philosophy by Bertrand Russell...", "language": "en", "duration": 612.4}
data: [DONE]
Because this matches OpenAI’s documented event shape, official OpenAI SDKs
work against this endpoint — point the client’s base_url at this service:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8010/v1", api_key="not-used")
with open("tests/philosophy_10_russell_128kb.mp3", "rb") as audio:
stream = client.audio.transcriptions.create(
model="whisper-1", file=audio, response_format="json", stream=True,
)
for event in stream:
print(event)
stream=truerequiresresponse_formatto bejsonorverbose_json;srt,vtt, andtextreturn HTTP 400.
Verify continuous audio streaming (WebSocket)#
WS /v1/realtime?intent=transcription accepts a live, continuous audio
feed. Clients push PCM16 frames and receive transcripts as each utterance
completes — server-side voice activity detection (VAD) decides where
utterances end.
Install a WebSocket client, decode some audio to raw PCM16, and stream it:
pip install websockets
# Decode 18s of audio to raw PCM16, mono, 16 kHz
ffmpeg -v error -i tests/philosophy_10_russell_128kb.mp3 -t 18 \
-f s16le -ar 16000 -ac 1 /tmp/audio.raw
import asyncio, base64, json, websockets
SAMPLE_RATE = 16000
async def main():
pcm = open("/tmp/audio.raw", "rb").read()
silence = b"\x00\x00" * int(SAMPLE_RATE * 1.2) # triggers VAD end-of-speech
url = "ws://127.0.0.1:8010/v1/realtime?intent=transcription"
async with websockets.connect(url, max_size=None) as ws:
print(json.loads(await ws.recv())["type"]) # transcription_session.created
async def send():
frame = int(SAMPLE_RATE * 0.1) * 2 # 100 ms frames
for i in range(0, len(pcm), frame):
await ws.send(json.dumps({
"type": "input_audio_buffer.append",
"audio": base64.b64encode(pcm[i:i + frame]).decode(),
}))
await asyncio.sleep(0.005)
await ws.send(json.dumps({
"type": "input_audio_buffer.append",
"audio": base64.b64encode(silence).decode(),
}))
asyncio.create_task(send())
while True:
msg = json.loads(await ws.recv())
print("EVENT:", msg["type"])
if msg["type"] == "conversation.item.input_audio_transcription.completed":
print("TRANSCRIPT:", msg["transcript"])
break
asyncio.run(main())
Expected event sequence:
transcription_session.created
input_audio_buffer.speech_started
input_audio_buffer.speech_stopped
input_audio_buffer.committed
conversation.item.input_audio_transcription.delta
conversation.item.input_audio_transcription.completed
TRANSCRIPT: Chapter X of Philosophy by Bertrand Russell...
If you never see speech_started, the audio is quieter than the VAD
threshold — lower it with a session.update message:
{"type": "session.update",
"session": {"turn_detection": {"threshold": 0.005, "silence_duration_ms": 700}}}
To disable VAD entirely and control utterance boundaries yourself, send
"turn_detection": null and then {"type": "input_audio_buffer.commit"}
whenever you want a transcript.
See the API Reference for the full event list and configuration options.
Verify VSS-compatible endpoints#
Video Search & Summarization (VSS) calls this service under an /api/v1
prefix:
curl --noproxy '*' http://127.0.0.1:8010/api/v1/models
Expected: a JSON body with models and default_model. The same routes are
also served unprefixed (/models, /transcriptions) for local use.
Run the automated tests#
pip install pytest httpx
pytest tests/test_streaming_endpoints.py tests/test_vss_endpoints.py -v
These cover the SSE event sequence, the realtime WebSocket handshake, VAD behavior, and the VSS contract, and require no model weights or GPU.
Next Steps#
API Reference for endpoint details and examples
Configuration Guide to customize models and devices
Troubleshooting for common startup issues