Deploy the vLLM Service for Defect Explanation#

This section shows how to deploy the multimodal sample application with the vLLM service enabled using the Makefile targets.

System Requirements#

Component

Minimum Requirement

Operating System

Ubuntu OS version 24.04 LTS or later

Hardware

Intel® Core™ Ultra Series 3 processor or newer

Prerequisites#

  1. Ensure the .env file is configured with valid values for:

    • HOST_IP

    • INFLUXDB_USERNAME, INFLUXDB_PASSWORD

    • VISUALIZER_GRAFANA_USER, VISUALIZER_GRAFANA_PASSWORD

    • MTX_WEBRTCICESERVERS2_0_USERNAME, MTX_WEBRTCICESERVERS2_0_PASSWORD

    • S3_STORAGE_USERNAME, S3_STORAGE_PASSWORD

Download Models#

This section shows how to download the Unsloth Qwen3.5-2B model and Unsloth Qwen3.5-2B fine-tuned LoRA adapter.

  1. Review and accept the Unsloth Qwen3.5-2B license before downloading.

    Note: The Low-Rank Adaptation (LoRA) adapter was specifically trained on a subset of the Intel Robotic Welding Multimodal Dataset and may not generalize to generic weld datasets.

  2. Go to the root folder of the multimodal sample app:

cd edge-ai-suites/manufacturing-ai-suite/industrial-edge-insights-multimodal

  1. Download the model and adapter:

mkdir -p configs/vllm/huggingface configs/vllm/models && \
cd configs/vllm/ && \
rm -rf .modelenv && \
python3 -m venv .modelenv && \
source .modelenv/bin/activate && \
pip3 install huggingface_hub==1.23.0 && \
rm -rf huggingface models && \
hf download unsloth/Qwen3.5-2B \
    --local-dir ./huggingface/Qwen3.5-2B && \
hf download Intel/qwen3.5-2b-vlm-weld-explainability-lora \
    --local-dir ./models/qwen3.5-2b-vlm-weld-explainability-lora && \
deactivate && \
cd ../..

Deploy the vLLM Service#

Note: vLLM preallocates GPU-addressable memory up to the limit specified by VLLM_GPU_MEMORY_UTILIZATION [Video Random Access Memory (VRAM) on discrete GPU (dGPU); shared system memory on integrated GPU (iGPU)]. Since the optimal value varies between platforms, update VLLM_GPU_MEMORY_UTILIZATION in the .env file to match your target hardware.

  1. Run the Makefile target:

 cd edge-ai-suites/manufacturing-ai-suite/industrial-edge-insights-multimodal
 make up_vllm

  1. (Optional) To build from source instead of deploying the pre-built artifacts, run the following for a fresh build before deployment:

cd edge-ai-suites/manufacturing-ai-suite/industrial-edge-insights-multimodal
make build
make up_vllm

Verify the Deployment#

  1. Check the overall stack health:

    Note: The command make status may show errors in containers like ia-grafana if you have not logged in to Grafana dashboard for the first time, or if your session has timed out. Log in to Grafana dashboard and if the dashboard works correctly, ignore user token not found and other minor errors in the Grafana logs.

    cd edge-ai-suites/manufacturing-ai-suite/industrial-edge-insights-multimodal
make status

  2. Confirm that the vLLM container is running:

    docker ps --filter "name=vllm-server"

  3. Inspect vLLM logs:

    docker logs -f vllm-server

  4. Check the output in Grafana dashboard:

    • Use the link https://localhost:3000 to open Grafana dashboard in a browser, preferably the Chrome browser. For Helm deployment, use the link https://localhost:30001.

    • Log in to Grafana dashboard using the VISUALIZER_GRAFANA_USER and VISUALIZER_GRAFANA_PASSWORD values from the .env file:

      Grafana login

    • After logging in, click Dashboards: Menu view

    • Select Multimodal Weld Defect Detection Explainability Dashboard: Multimodal Weld Defect Detection Explainability Dashboard

    • The following appears:

      vLLM Reasoning for weld data

Stop the Deployment#

To bring down the full stack:

cd edge-ai-suites/manufacturing-ai-suite/industrial-edge-insights-multimodal
make down

Troubleshooting#

  • vllm-server startup delay after deployment The vllm-server service can take about 10 minutes to fully come up after make up_vllm. This is expected while the model is initialized and loaded into memory.

  • Error: configs/vllm/models directory does not exist. Create the directory and place the required model artifacts in it.

  • Error: configs/vllm/models directory is empty. Add model files and checkpoints before running make up_vllm.

  • HOST_IP is not set or HOST_IP is not a valid IPv4 address format. Update HOST_IP in .env with a valid IPv4 address.

  • Username and password validation failures from check_env_variables Update .env values so they match the Makefile validation rules.