# Get Started Clone the repository, build the stack, and confirm a working voice ordering session end to end. Confirm your machine meets the [System Requirements](./get-started/system-requirements.md) before starting. ## Application Overview Smart Kiosk Assistant is a voice-driven quick-service-restaurant (QSR) ordering kiosk. The browser captures microphone audio and uploads it to `audio-analyzer` (Whisper ASR + speaker diarization); `rag-service` runs the RAG pipeline and ordering agent on a Qwen3-4B model served by `ovms-llm`; `text-to-speech` synthesizes the spoken reply. `kiosk-core` orchestrates the session and product ordering, `kiosk-ui` is the React front end, and `rtsp-streamer` + `queue-service` provide person-counting queue analytics. `metrics-collector` reports hardware utilization, and the optional `identity-service` adds face + voice authentication. ## Prerequisites - Ubuntu 22.04 or newer (Linux, Desktop edition — a browser with microphone access is required for voice ordering) - [Docker](https://docs.docker.com/engine/install/) 24.0+ - [Docker Compose](https://docs.docker.com/compose/install/) V2+ - [Make](https://www.gnu.org/software/make/) (`sudo apt install make`) - Intel hardware (CPU, iGPU, or NPU — see [Device support](#device-support-cpu-gpu-npu) before choosing NPU) - Intel GPU drivers (recommended) — see the [Intel GPU driver guide](https://dgpu-docs.intel.com) - A [Hugging Face account and access token](https://huggingface.co/settings/tokens), with the three gated Pyannote licenses accepted — required for speaker diarization: [`speaker-diarization-community-1`](https://huggingface.co/pyannote/speaker-diarization-community-1), [`speaker-diarization-3.1`](https://huggingface.co/pyannote/speaker-diarization-3.1), and [`segmentation-3.0`](https://huggingface.co/pyannote/segmentation-3.0) - At least 30 GB free disk space for models and Docker images > [!NOTE] > First-time setup downloads the OpenVINO™ Model Server (OVMS) LLM model (~4 GB) plus the > in-container ASR, TTS, embedding, and reranker models. Allow extra time on > the first run depending on your connection. ## Device support (CPU, GPU, NPU) CPU and iGPU are supported everywhere in the stack. **NPU is only partially supported, and choosing it without checking the constraints below produces a deployment that either crash-loops or silently runs on CPU.** Read this before setting `TARGET_DEVICE` in [Step 3](#step-3-create-env-and-set-required-variables). | Component | CPU | GPU | NPU | | --- | --- | --- | --- | | `ovms-llm` (served LLM) | ✅ | ✅ | ❌ **Not supported** | | `audio-analyzer` ASR | ✅ | ✅ | ⚠️ `whisper-tiny` / `whisper-base` only | | `queue-service`, `identity-service` | ✅ | ✅ | ✅ | - **Do not set `TARGET_DEVICE=NPU` for the LLM.** OVMS serves the NPU through a Stateful servable that caps prompts at 1024 tokens, so LLM turns fail with "Sorry, I encountered an error". `setup_models.sh --device NPU` refuses to run for this reason and exits with an error. Use `GPU` (or `CPU`) instead. - **NPU is usable for the non-LLM models.** If you only want the queue, identity and ASR models on the NPU, pass `--skip-ovms` to acknowledge that the LLM is not being prepared for NPU: ```bash ./setup_models.sh --device NPU --skip-ovms ``` The OVMS LLM must still be served on `GPU` or `CPU`. - **ASR on NPU needs a small checkpoint.** `whisper-small` and larger fail to compile with `Check '!self_attn_nodes.empty()' failed`. - **NPU passthrough requires `ACCEL_MOUNT_PATH`** in `.env`. Without it the container has no accel node, OpenVINO silently falls back to CPU, and the run is mislabelled rather than failed. See [Troubleshooting](troubleshooting.md) for the full diagnostics and fixes. ## Step 1: Install Docker and Intel GPU drivers Skip this step if Docker is already installed (`docker --version` and `docker compose version` both succeed). Otherwise, install Docker Engine and Docker Compose v2: ```bash # Docker Engine + Compose v2 sudo apt-get update sudo apt-get install -y ca-certificates curl gnupg lsb-release sudo install -m 0755 -d /etc/apt/keyrings curl -fsSL https://download.docker.com/linux/ubuntu/gpg \ | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] \ https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" \ | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null sudo apt-get update sudo apt-get install -y docker-ce docker-ce-cli containerd.io \ docker-buildx-plugin docker-compose-plugin sudo usermod -aG docker $USER && newgrp docker ``` For Intel iGPU acceleration (recommended), install the GPU compute runtime: ```bash sudo apt-get install -y intel-opencl-icd intel-level-zero-gpu level-zero sudo usermod -aG render $USER && newgrp render ``` Verify the GPU device node exists: `ls /dev/dri/renderD*`. If the packages are not found, follow the [Intel GPU driver guide](https://dgpu-docs.intel.com). Skip this block and set `TARGET_DEVICE=CPU` in `.env` if no Intel GPU is available. ## Step 2: Clone the Repository ```bash git clone --recurse-submodules -b --single-branch https://github.com/intel-retail/voice-enabled-interactions.git #e.g. v4.0.0 cd voice-enabled-interactions/smart-kiosk-assistant ``` ## Step 3: Create `.env` and Set Required Variables Create your environment file from the template: ```bash make init-env # copies .env.example → .env ``` Open `.env` and set your Hugging Face token (free account at ): ```bash # .env HF_TOKEN=hf_your_token_here ``` Speaker diarization uses three **gated** Pyannote models. While signed in to Hugging Face with the same account that owns the token, accept the license on **all three** model pages — accepting only some of them is not enough, and the `audio-analyzer` container will fail to download the pipeline: - - - > [!NOTE] > Diarization is off by default (`KIOSK_CORE_DIARIZATION_ENABLED=false`), so > these licenses are needed only if you set it to `true`. With it off, > speech-to-text works normally; only per-speaker attribution is unavailable. For Intel GPU inference (the default `TARGET_DEVICE=GPU`), set `RENDER_GID` so containers can access `/dev/dri`: ```bash # find the render group id on your host getent group render | cut -d: -f3 # then set it in .env, e.g. RENDER_GID=992 ``` Set `TARGET_DEVICE=CPU` in `.env` if no Intel GPU is available. ## Step 4: Download the LLM Model for OVMS The ordering agent uses a Qwen3-4B model served by OVMS. Download it before starting the stack: ```bash # GPU (recommended, INT8 ~4 GB) ./setup_models.sh # CPU only (no Intel GPU) ./setup_models.sh --device CPU # INT4 model (smaller/faster, ~2 GB, slightly lower accuracy) ./setup_models.sh --int4 ``` The script downloads the model, pre-converted for the OpenVINO™ Toolkit (OpenVINO™), into `./models/` and updates `OVMS_MODEL_NAME`, `TARGET_DEVICE`, and `RENDER_GID` in `.env`. The download happens once — subsequent starts reuse the cached model. ## Step 5: Download Sample Videos for Queue Analytics The `queue-service` person-counting pipeline consumes an RTSP stream published by `rtsp-streamer` from clips under `Sample_data/`. Set the download URLs in `.env` (`SAMPLE_VIDEO_1_URL`, `SAMPLE_VIDEO_2_URL`), then fetch them: ```bash make download-sample-video ``` The clips are saved as `Sample_data/sample_1.mp4` and `Sample_data/sample_2.mp4` to match the `MEDIA_FILES` entry in `docker-compose.yml`. You can also copy your own MP4 files to those paths manually. To switch the RTSP streamer between the bundled sample clips and a live webcam, edit `.env` only and restart the stack: ```bash RTSP_SOURCE_MODE=sample # or live CAMERA_DEVICE=/dev/video0 ``` `sample` mode uses the existing `MEDIA_FILES` / `MEDIA_FILE` flow and publishes the sample clips continuously. `live` mode uses the configured `CAMERA_DEVICE` and keeps the RTSP endpoint unchanged at `rtsp://rtsp-streamer:8554/queue`. ### Live camera streaming To stream a webcam instead of the bundled sample clips, set the RTSP source mode to `live` in `.env` and choose the device node that matches your camera: ```bash RTSP_SOURCE_MODE=live CAMERA_DEVICE=/dev/video2 ``` If you are not sure which device node to use, list the available camera devices on the host: ```bash ls /dev/video* ``` If `v4l2-ctl` is installed, you can also run: ```bash v4l2-ctl --list-devices ``` Update `.env` with the correct `CAMERA_DEVICE` value, then restart the RTSP streamer or restart Docker Compose so the new configuration is picked up: ```bash docker compose up -d rtsp-streamer ``` The RTSP endpoint stays the same in both modes: ```text rtsp://rtsp-streamer:8554/queue ``` ## Step 6: Build Images and Start the Stack ```bash make build # pull released images (shipped .env has REGISTRY=true) make up # start the stack (runs check-env first) ``` The equivalent raw Compose commands are: ```bash docker compose build docker compose up -d ``` With the shipped `.env` (`REGISTRY=true`), `make build` **pulls** the released service images from the registry rather than compiling them. To build the local service images from source instead, set `REGISTRY=false` (`make build REGISTRY=false`); `ovms-llm` always uses a pre-built upstream image and is pulled automatically. A from-source build takes 10–20 minutes the first time. > [!NOTE] > > **`REGISTRY` and `REGISTRY_REPO` are two different settings.** They are easy > to confuse because of the similar names: > > | Variable | Shipped value | Read by | Meaning | > | --- | --- | --- | --- | > | `REGISTRY` | `true` | the `Makefile` only | Build mode: `true` = pull released images, `false` = build from source | > | `REGISTRY_REPO` | `intel` | `docker-compose.yml` | Image **name prefix**, e.g. `intel/kiosk-core:2026.3.0-rc2` | > > Every `image:` entry in `docker-compose.yml` resolves through > `${REGISTRY_REPO:-intel}`, never through `REGISTRY`. Keep `REGISTRY_REPO=intel` > unless you publish these images under your own namespace — setting it to a > `true`/`false` value makes Compose try to pull from a registry account literally > named `true` and every pull fails. Change `REGISTRY` only to switch between > pulling and building. ### Which image version you get: `RELEASE_TAG` The shipped `.env` sets `RELEASE_TAG=2026.3.0-rc2` (the current release), and every `image:` entry resolves to `${REGISTRY_REPO}/:${RELEASE_TAG}`. With `REGISTRY=true`, `make build` therefore **pulls** the images published for that release tag rather than building the code in your checkout. This matters when you are verifying a fix. Pulling a newer repository revision does **not** by itself change the `audio-analyzer`, `text-to-speech`, `rag-service`, `kiosk-core`, `kiosk-ui`, `rtsp-streamer` or `queue-service` containers you run — those come from the registry. A source change only reaches them once the published image is rebuilt and re-tagged. If you are reproducing or validating a specific fix, pick one of these: ```bash # A. Pin a known release tag (reproducible, recommended for validation) RELEASE_TAG=2026.3.0-rc2 # in .env (any published tag) make build make up # B. Build the local services from your checkout instead of pulling make build REGISTRY=false make up ``` Option B builds `kiosk-core`, `rag-service`, `kiosk-ui`, `rtsp-streamer` and `queue-service` from source (10–20 minutes the first time). `audio-analyzer` and `text-to-speech` live in the sibling `edge-ai-libraries` repository, so `REGISTRY=false` does not rebuild them: an image already present locally under the selected registry and tag is kept as-is (it may be your own source build from `edge-ai-libraries`), and only a missing one is pulled. `ovms-llm` is always an upstream image. To confirm what you are actually running — this reads the live container, so it stays correct whichever registry and tag you selected: ```bash make show-config # prints REGISTRY, the compose image prefix, and RELEASE_TAG docker inspect kiosk-core --format '{{.Config.Image}} (image ID: {{.Image}})' ``` > `RELEASE_TAG` in `.env` must match the Makefile default (`RELEASE_TAG ?= 2026.3.0-rc2`) > unless you deliberately pin a different one. `METRICS_COLLECTOR_TAG` is > independent and is versioned separately. The default stack starts ten containers (`QUEUE=true` by default; identity is off): | Container | Port | Role | | --- | --- | --- | | `ovms-llm` | 8000 (REST), 9001 (gRPC) | Serves Qwen3-4B via OpenAI-compatible API | | `metrics-collector` | 9000 | Hardware utilization metrics | | `audio-analyzer` | 8010 | Whisper ASR + speaker diarization | | `text-to-speech` | 8011 | Kokoro TTS synthesis | | `rag-service` | 8020 | RAG pipeline + ordering agent | | `kiosk-core` | 8012 | Session API + product ordering | | `kiosk-ui` | 7860 | Voice kiosk React UI (operator view) | | `kiosk-ui-customer` | 7861 | Voice kiosk React UI (customer view) | | `rtsp-streamer` | 8554 | Publishes sample clips as an RTSP stream | | `queue-service` | 8090 | Person detection + queue counting | `rtsp-streamer` and `queue-service` start only while `QUEUE=true` (the shipped default); set `QUEUE=false` to leave them out and free CPU for voice runs. The optional `identity-service` (port `8013`, face + voice authentication) starts only when identity is enabled: ```bash make up IDENTITY=true ``` ## Step 7: Verify the Stack Is Healthy Services start in dependency order. Allow 2–5 minutes on first run for model assets to download into Docker volumes. ```bash make status # container status (or: docker compose ps) make test # health-check ovms, audio, tts, rag, kiosk-core, kiosk-ui make logs # follow logs for all services (or make logs-core, make logs-rag, ...) ``` You can also probe individual endpoints directly: ```bash curl --noproxy '*' http://127.0.0.1:8000/v3/models # ovms-llm curl --noproxy '*' http://127.0.0.1:8010/health # audio-analyzer curl --noproxy '*' http://127.0.0.1:8011/health # text-to-speech curl --noproxy '*' http://127.0.0.1:8020/health # rag-service curl --noproxy '*' http://127.0.0.1:8012/health # kiosk-core curl --noproxy '*' http://127.0.0.1:8090/health # queue-service ``` Every health endpoint should return `{"status": "ok"}`. The OVMS endpoint returns the active model name (`OpenVINO/Qwen3-4B-int8-ov`). ## Step 8: Open the Kiosk and Try Voice Ordering Open a browser on the same machine: ```text http://127.0.0.1:7860 ``` 1. Click **Allow** when the browser asks for microphone permission. (Use `127.0.0.1`, not the machine hostname — browsers block microphone on non-HTTPS origins except `localhost`.) 2. Click the **🎤 microphone** button and speak. Try: - *"What's on the menu?"* - *"Show me your burgers"* - *"I'd like to order a Classic Chicken Burger"* - *"Confirm my order"* 3. Watch the **🍔 QSR** tab in the right panel — the **Menu** sub-tab shows the full catalog; the **Cart** sub-tab shows your live order and the confirmed receipt after you say *"Confirm"*. ## Stop the Services ```bash make down # stop all containers (or: docker compose down) make clean # stop and remove named volumes for a clean restart ``` ## Quick Start Reference | Task | Command | Description | | --- | --- | --- | | Create environment file | `make init-env` | Copy `.env.example` → `.env` | | Download LLM model | `make setup-models` | Fetch the OVMS Qwen3-4B model | | Download sample videos | `make download-sample-video` | Fetch queue-service RTSP clips | | Build images | `make build` | Build locally (shipped .env has `REGISTRY=true`) | | Start services | `make up` | Start the full stack | | Verify health | `make test` | Check all service endpoints | | View logs | `make logs` | Tail logs for all services | | Stop services | `make down` | Stop all containers | | Clean restart | `make clean` | Stop and remove named volumes | ## Next Steps - [How It Works](./how-it-works.md) - [Configuration](./get-started/configuration.md) - [Build From Source](./get-started/build-from-source.md) - [API Reference](./api-reference.md) - [Troubleshooting](./troubleshooting.md) :::{toctree} :hidden: ./get-started/system-requirements.md ./get-started/build-from-source.md ./get-started/run-container.md ./get-started/configuration.md :::