Handheld Multi-Modal Application Deployment#

This guide provides instructions on how to deploy the Handheld Multi-Modal application on a local machine.

(Optional) Configuring the Proxy#

Note: If not using proxy servers, skip to Deploying the Application.

Depending on the system’s network configuration, you may need an additional proxy configuration. Ensure that /etc/environment contains proxy variables; replace proxy-example:123 with a valid proxy for the local environment:

sudo tee -a /etc/environment > /dev/null <<EOF
http_proxy="http://proxy-example:123"
https_proxy="http://proxy-example:123"
ftp_proxy="http://proxy-example:123"
no_proxy="localhost,127.0.0.1,10.0.0.0/8,192.0.0.0/8,fedaero.intel.com,vippet,grafana,metrics-manager"
EOF

source /etc/environment

Configure the proxy for the Docker client:

mkdir -p ~/.docker
tee -a ~/.docker/config.json > /dev/null <<EOF
{
    "proxies": {
        "default": {
            "httpProxy":  "http://proxy-example:123",
            "httpsProxy": "http://proxy-example:123",
            "noProxy":    "localhost,127.0.0.1,10.0.0.0/8,192.0.0.0/8,fedaero.intel.com,vippet,grafana,metrics-manager"
        }
    }
}
EOF

Configure the proxy for Docker containers:

sudo mkdir -p /etc/systemd/system/docker.service.d
sudo tee /etc/systemd/system/docker.service.d/http-proxy.conf > /dev/null <<EOF
[Service]
Environment="HTTP_PROXY=http://proxy-example:123"
Environment="HTTPS_PROXY=http://proxy-example:123"
Environment="NO_PROXY=localhost,127.0.0.1,10.0.0.0/8,192.0.0.0/8,fedaero.intel.com,vippet,grafana,metrics-manager"
EOF

Restart the Docker daemon:

sudo systemctl daemon-reload
sudo systemctl restart docker

Verify the Docker daemon’s proxy configurations (sample output below):

docker info|grep -i PROXY
 HTTP Proxy: http://proxy-example:123
 HTTPS Proxy: http://proxy-example:123
 No Proxy: localhost,127.0.0.1,10.0.0.0/8,192.0.0.0/8,fedaero.intel.com,vippet,grafana,metrics-manager

Deploying the Application#

Download the compressed file:

curl -OjL https://github.com/open-edge-platform/edge-ai-suites/releases/download/fedaero-latest/handheld-multi-modal.zip

Decompress the downloaded file:

unzip handheld-multi-modal.zip

Run the script that installs all dependencies, downloads models, and starts applications. During installation, a single prompt asking to accept licenses of models will appear. Depending on network bandwidth, it takes around 10-15 minutes. If an error occurs during installation, see the proxy configuration step:

cd handheld-multi-modal
./run.sh up

Verifying the installation#

After the script finishes, verify that the containers are running (sample output below):

docker ps
CONTAINER ID   IMAGE                                                   COMMAND                  CREATED          STATUS                             PORTS                                                                                                                                   NAMES
45aeb6ad8884   nginx:alpine                                            "/docker-entrypoint.…"   27 seconds ago   Up 25 seconds                      127.0.0.1:443->443/tcp, 127.0.0.1:5443->5443/tcp, 127.0.0.1:7443->7443/tcp, 80/tcp, 127.0.0.1:8443->8443/tcp                            nginx-https
1cf974e6c425   ghcr.io/open-webui/open-webui:v0.11.0-slim               "bash start.sh"          27 seconds ago   Up 25 seconds (health: starting)   8080/tcp                                                                                                                                open-webui
90c0db070f36   whisper-stt:latest                                      "/entrypoint.sh pyth…"   27 seconds ago   Up 26 seconds                      5000/tcp                                                                                                                                whisper-stt
ee1cef103480   grafana/grafana:13.1.0-25893932881                      "/run.sh"                27 seconds ago   Up 26 seconds                      3000/tcp                                                                                                                                grafana
231fd29c88d8   openvino/model_server:latest-gpu                        "/ovms/bin/ovms --re…"   27 seconds ago   Up 26 seconds                                                                                                                                                              ovms
3dc8dfefa60e   intel/vippet-ui:2026.1.0-20260512-weekly                "/docker-entrypoint.…"   34 seconds ago   Up 27 seconds                      0.0.0.0:80->80/tcp, [::]:80->80/tcp                                                                                                     ui
d1ec3f394245   intel/vippet-app:2026.1.0-20260512-weekly               "./entrypoint.sh"        34 seconds ago   Up 33 seconds (healthy)            0.0.0.0:7860->7860/tcp, [::]:7860->7860/tcp                                                                                             vippet
9fa7733f0cc4   bluenviron/mediamtx:1.15.6                              "/mediamtx"              34 seconds ago   Up 33 seconds                      0.0.0.0:8554->8554/tcp, [::]:8554->8554/tcp, 0.0.0.0:8189->8189/udp, [::]:8189->8189/udp, 0.0.0.0:8889->8889/tcp, [::]:8889->8889/tcp   mediamtx
76d9c62a039b   intel/vippet-onvif-discovery:2026.1.0-20260512-weekly   "/bin/sh -c 'python …"   34 seconds ago   Up 33 seconds                                                                                                                                                              onvif-discovery
f9d9fc705f29   intel/metrics-manager:2026.1.0-20260508-weekly          "/entrypoint.sh"         34 seconds ago   Up 33 seconds (healthy)            0.0.0.0:9090->9090/tcp, [::]:9090->9090/tcp, 8186/tcp, 0.0.0.0:9273->9273/tcp, [::]:9273->9273/tcp                                      metrics-manager
c7e676f86e1b   intel/model-download:2026.1.0-20260505-weekly           "/opt/entrypoint.sh …"   34 seconds ago   Up 33 seconds (healthy)            0.0.0.0:8000->8000/tcp, [::]:8000->8000/tcp

Note: After a system restart, run ./run up from the handheld-multi-modal directory to start the applications again.

Accessing Application User Interface#

This composite application exposes multiple endpoints through the NGINX TLS reverse proxy. They are bound to localhost only and are not exposed on any external IP address. Since the intended use is on handheld devices, the applications do not provide authentication or authorization.

Notice: The “self-signed certificate” browser warning is expected. Modern browsers require HTTPS to enable microphone input used by Open WebUI and Speech To Text services, therefore, the NGINX reverse proxy uses the certificate to ensure TLS transport on the localhost bound addresses.

To add a certificate to the trust pool, open a browser of your choice and navigate to Settings->Certificates->Manage certificates page. Next, select the Authorities tab and click Import. In file explorer, navigate to the folder that contains Handheld Multi-Modal Application and select the data/nginx-certs/ca-cert.pem file. Finally, select the Trust this CA to identify websites checkbox and click Ok.

Service	URL	Notes
Single pane page	https://localhost:443	via NGINX reverse proxy
Visual Pipeline and Platform Evaluation Tool UI	https://localhost:1443	via NGINX reverse proxy
Open WebUI	https://localhost:8443	Conversational Agent backed by LLM — browser microphone enabled (via NGINX reverse proxy)
Whisper speech-to-text service	https://localhost:5443	Speech-to-text — browser microphone enabled (via NGINX reverse proxy)
Grafana dashboard	https://localhost:7443	Pre-provisioned dashboards (via NGINX reverse proxy)

ViPPET#

After opening the main page of ViPPET, you can select one of the several options available in ViPPET:

Pipelines - to check and run one of the predefined pipelines and understand how it works. Predefined pipelines will provide a video output stream that shows how a model works, as well as a small subset of system metrics during execution of the pipeline. It also exposes an option to add a new custom pipeline for advanced users, but it might require additional input from other options.
Benchmarks - allows you to compare performance of pipelines on different combinations of supported hardware (CPU/NPU/GPU/NPU+GPU) to find out the best device for a specific pipeline.
Models - exposes a list of available models that can be used in a pipeline, as well as the import functionality for models.
Video/Images/Cameras - allows configuration of data source that could be used in a pipeline.

Open WebUI#

The main page exposes a chat with the default AI model, where you can ask questions. If an initial response is not sufficient, the chat with the AI model can be continued in the same window, which will keep context. If a question is related to attachment(s), such as a file, web-page or other chat, it could be added by clicking the + sign and selecting the corresponding option. It is recommended to open the New Chat window and start a new chat to change the topic and to keep the AI model context clean, as it improves results.

Whisper#

On the main page of Whisper, there are two options to upload audio for the transcription:

By uploading an audio file in one of the supported audio formats (“flac”, “m4a”, “mp3”, “mp4”, “ogg”, “wav”, “webm”) and clicking on the corresponding section in UI, and selecting a file in the file explorer, or by drag-and-dropping the file into the section.
By recording audio from a microphone. Click the round record button, allow Whisper access to the microphone and start talking into the microphone. In both cases, the transcription text will appear on the lower part of the page in real-time, in parts, as soon as Whisper has completed the transcription of a part of the recording. Once the whole recording has been transcribed, an additional section will appear and show the duration and ratio of the transcription.

Grafana#

By default, the main page shows generic metrics from the system, such as CPU/NPU and power consumption. To switch to more detailed, per-application view, navigate to Dashboards->Panther Lake Live Dashboard. On this dashboard, most of the metrics will be either empty or will not display any values, since the metrics are gathered in real-time and other applications must execute a workload to generate metrics. The metrics map to other handheld applications as follows:

Frame Rate Over Time and Latest Pipeline Frame Rate Average are from ViPPET.
LLM Number of Responses Generated and LLM Number of Responses Generated are from Open WebUI.
Speech to text Processing Ratio Last and Speech To Text Processing Ratio are from Whisper. Metrics do not persist in any database, so a refresh of the page will reset ALL gathered metrics.