# Handheld Multi-Modal Application Deployment

This guide provides instructions on how to deploy the Handheld Multi-Modal application on a
local machine.

## (Optional) Configuring the Proxy

> **Note**: If not using proxy servers, skip to [Deploying the Application](#deploying-the-application).

Depending on the system's network configuration, you may need an additional proxy configuration.
Ensure that `/etc/environment` contains proxy variables; replace `proxy-example:123` with a
valid proxy for the local environment:

```bash
sudo tee -a /etc/environment > /dev/null <<EOF
http_proxy="http://proxy-example:123"
https_proxy="http://proxy-example:123"
ftp_proxy="http://proxy-example:123"
no_proxy="localhost,127.0.0.1,10.0.0.0/8,192.0.0.0/8,fedaero.intel.com,vippet,grafana,metrics-manager"
EOF

source /etc/environment
```

Configure the proxy for the Docker client:

```bash
mkdir -p ~/.docker
tee -a ~/.docker/config.json > /dev/null <<EOF
{
    "proxies": {
        "default": {
            "httpProxy":  "http://proxy-example:123",
            "httpsProxy": "http://proxy-example:123",
            "noProxy":    "localhost,127.0.0.1,10.0.0.0/8,192.0.0.0/8,fedaero.intel.com,vippet,grafana,metrics-manager"
        }
    }
}
EOF
```

Configure the proxy for Docker containers:

```bash
sudo mkdir -p /etc/systemd/system/docker.service.d
sudo tee /etc/systemd/system/docker.service.d/http-proxy.conf > /dev/null <<EOF
[Service]
Environment="HTTP_PROXY=http://proxy-example:123"
Environment="HTTPS_PROXY=http://proxy-example:123"
Environment="NO_PROXY=localhost,127.0.0.1,10.0.0.0/8,192.0.0.0/8,fedaero.intel.com,vippet,grafana,metrics-manager"
EOF
```

Restart the Docker daemon:

```bash
sudo systemctl daemon-reload
sudo systemctl restart docker
```

Verify the Docker daemon's proxy configurations (sample output below):

```text
docker info|grep -i PROXY
 HTTP Proxy: http://proxy-example:123
 HTTPS Proxy: http://proxy-example:123
 No Proxy: localhost,127.0.0.1,10.0.0.0/8,192.0.0.0/8,fedaero.intel.com,vippet,grafana,metrics-manager
```

## Deploying the Application

Download the compressed file:

```bash
curl -OjL https://github.com/open-edge-platform/edge-ai-suites/releases/download/fedaero-latest/handheld-multi-modal.zip
```

Decompress the downloaded file:

```bash
unzip handheld-multi-modal.zip
```

Run the script that installs all dependencies, downloads models, and starts applications.
During installation, a single prompt asking to accept licenses of models will appear.
Depending on network bandwidth, it takes around 10-15 minutes. If an error occurs during
installation, see the [proxy configuration step](#optional-configure-the-proxy):

```bash
cd handheld-multi-modal
./run.sh up
```

## Verifying the installation

After the script finishes, verify that the containers are running (sample output below):

```text
docker ps
CONTAINER ID   IMAGE                                                   COMMAND                  CREATED          STATUS                             PORTS                                                                                                                                   NAMES
45aeb6ad8884   nginx:alpine                                            "/docker-entrypoint.…"   27 seconds ago   Up 25 seconds                      127.0.0.1:443->443/tcp, 127.0.0.1:5443->5443/tcp, 127.0.0.1:7443->7443/tcp, 80/tcp, 127.0.0.1:8443->8443/tcp                            nginx-https
1cf974e6c425   ghcr.io/open-webui/open-webui:v0.11.0-slim               "bash start.sh"          27 seconds ago   Up 25 seconds (health: starting)   8080/tcp                                                                                                                                open-webui
90c0db070f36   whisper-stt:latest                                      "/entrypoint.sh pyth…"   27 seconds ago   Up 26 seconds                      5000/tcp                                                                                                                                whisper-stt
ee1cef103480   grafana/grafana:13.1.0-25893932881                      "/run.sh"                27 seconds ago   Up 26 seconds                      3000/tcp                                                                                                                                grafana
231fd29c88d8   openvino/model_server:latest-gpu                        "/ovms/bin/ovms --re…"   27 seconds ago   Up 26 seconds                                                                                                                                                              ovms
3dc8dfefa60e   intel/vippet-ui:2026.1.0-20260512-weekly                "/docker-entrypoint.…"   34 seconds ago   Up 27 seconds                      0.0.0.0:80->80/tcp, [::]:80->80/tcp                                                                                                     ui
d1ec3f394245   intel/vippet-app:2026.1.0-20260512-weekly               "./entrypoint.sh"        34 seconds ago   Up 33 seconds (healthy)            0.0.0.0:7860->7860/tcp, [::]:7860->7860/tcp                                                                                             vippet
9fa7733f0cc4   bluenviron/mediamtx:1.15.6                              "/mediamtx"              34 seconds ago   Up 33 seconds                      0.0.0.0:8554->8554/tcp, [::]:8554->8554/tcp, 0.0.0.0:8189->8189/udp, [::]:8189->8189/udp, 0.0.0.0:8889->8889/tcp, [::]:8889->8889/tcp   mediamtx
76d9c62a039b   intel/vippet-onvif-discovery:2026.1.0-20260512-weekly   "/bin/sh -c 'python …"   34 seconds ago   Up 33 seconds                                                                                                                                                              onvif-discovery
f9d9fc705f29   intel/metrics-manager:2026.1.0-20260508-weekly          "/entrypoint.sh"         34 seconds ago   Up 33 seconds (healthy)            0.0.0.0:9090->9090/tcp, [::]:9090->9090/tcp, 8186/tcp, 0.0.0.0:9273->9273/tcp, [::]:9273->9273/tcp                                      metrics-manager
c7e676f86e1b   intel/model-download:2026.1.0-20260505-weekly           "/opt/entrypoint.sh …"   34 seconds ago   Up 33 seconds (healthy)            0.0.0.0:8000->8000/tcp, [::]:8000->8000/tcp
```

> **Note**: After a system restart, run `./run up` from the `handheld-multi-modal` directory to start the applications again.

## Accessing Application User Interface

This composite application exposes multiple endpoints through the NGINX TLS reverse proxy.
They are bound to localhost only and are not exposed on any external IP address.
Since the intended use is on handheld devices, the applications do not provide authentication
or authorization.

> **Notice**:
> The "self-signed certificate" browser warning is expected.
> Modern browsers require HTTPS to enable microphone input used by Open WebUI and
  Speech To Text services, therefore, the NGINX reverse proxy uses the certificate to ensure
  TLS transport on the `localhost` bound addresses.

To add a certificate to the trust pool, open a browser of your choice and navigate to `Settings->Certificates->Manage certificates` page. Next, select the `Authorities` tab and click `Import`. In file explorer, navigate to the folder that contains Handheld Multi-Modal Application and select the `data/nginx-certs/ca-cert.pem` file. Finally, select the `Trust this CA to identify websites` checkbox and click `Ok`.


| Service | URL | Notes |
|---------|-----|-------|
| Single pane page | https://localhost:443 | via NGINX reverse proxy |
| Visual Pipeline and Platform Evaluation Tool UI | https://localhost:1443 | via NGINX reverse proxy |
| Open WebUI | https://localhost:8443 | Conversational Agent backed by LLM — browser microphone enabled (via NGINX reverse proxy) |
| Whisper speech-to-text service | https://localhost:5443 | Speech-to-text — browser microphone enabled (via NGINX reverse proxy) |
| Grafana dashboard | https://localhost:7443 | Pre-provisioned dashboards (via NGINX reverse proxy) |


<!--
Source: [Endpoints](https://github.com/open-edge-platform/edge-ai-suites/blob/release-2026.2.0/federal-and-aerospace-ai-suite/handheld-multi-modal/README.md#endpoints)
-->

### ViPPET

After opening the main page of ViPPET, you can select one of the several options available in ViPPET:
1) Pipelines - to check and run one of the predefined pipelines and understand how it works. Predefined pipelines will provide a video output stream that shows how a model works, as well as a small subset of system metrics during execution of the pipeline. It also exposes an option to add a new custom pipeline for advanced users, but it might require additional input from other options. 
2) Benchmarks - allows you to compare performance of pipelines on different combinations of supported hardware (CPU/NPU/GPU/NPU+GPU) to find out the best device for a specific pipeline.
3) Models - exposes a list of available models that can be used in a pipeline, as well as the import functionality for models.
4) Video/Images/Cameras - allows configuration of data source that could be used in a pipeline.

### Open WebUI

The main page exposes a chat with the default AI model, where you can ask questions. If an initial response is not sufficient, the chat with the AI model can be continued in the same window, which will keep context. If a question is related to attachment(s), such as a file, web-page or other chat, it could be added by clicking the `+` sign and selecting the corresponding option.
It is recommended to open the `New Chat` window and start a new chat to change the topic and to keep the AI model context clean, as it improves results.

### Whisper

On the main page of Whisper, there are two options to upload audio for the transcription:
1) By uploading an audio file in one of the supported audio formats ("flac", "m4a", "mp3", "mp4", "ogg", "wav", "webm") and clicking on the corresponding section in UI, and selecting a file in the file explorer, or by drag-and-dropping the file into the section.
2) By recording audio from a microphone. Click the round `record` button, allow Whisper access to the microphone and start talking into the microphone.
In both cases, the transcription text will appear on the lower part of the page in real-time, in parts, as soon as Whisper has completed the transcription of a part of the recording. Once the whole recording has been transcribed, an additional section will appear and show the duration and ratio of the transcription.

### Grafana

By default, the main page shows generic metrics from the system, such as CPU/NPU and power consumption. To switch to more detailed, per-application view, navigate to `Dashboards->Panther Lake Live Dashboard`. On this dashboard, most of the metrics will be either empty or will not display any values, since the metrics are gathered in real-time and other applications must execute a workload to generate metrics. The metrics map to other handheld applications as follows:
1) `Frame Rate Over Time` and `Latest Pipeline Frame Rate Average` are from ViPPET.
2) `LLM Number of Responses Generated` and `LLM Number of Responses Generated` are from Open WebUI.
3) `Speech to text Processing Ratio Last` and `Speech To Text Processing Ratio` are from Whisper.
Metrics do not persist in any database, so a refresh of the page will reset ALL gathered metrics.