Change the Captioning (VLM) Model#

Each camera’s scene captions come from a vision-language model run by the gvagenai element. The model is chosen per camera in configs/cameras-pymavlink.yaml under captioning.model; no pipeline or compose edit is needed to switch. This page covers the bundled model, the format a replacement must be in, how to download or export one, and how to apply it. For the detection model, see Modify the Detection Model.

The bundled model#

make vlm-model downloads InternVL2-1B (OpenGVLab/InternVL2-1B) and converts it to OpenVINO™ IR, INT8 weights, for the CPU and the GPU:

resources/ov_models/
├── cpu/InternVL2-1B/
└── gpu/InternVL2-1B/

The VLM runs on the CPU or the GPU only (not the NPU). The conversion uses the shared download_models.sh from the Live Video Captioning sample application, fetched once into .cache/.

How a camera selects the model#

cameras:
  - id: front_camera
    captioning:
      enabled: true
      device: gpu                    # cpu | gpu
      model: InternVL2-1B            # folder name under resources/ov_models/<device>/
      prompt: Describe what you see in the scene.
      max_new_tokens: 50

model is a folder name; the manager resolves it to /home/pipeline-server/resources/ov_models/<device>/<model> and passes it to the pipeline as captioner_model_path. The folder must exist for the device the camera uses.

Required format#

gvagenai loads the model with openvino_genai.VLMPipeline, so the folder must contain a complete OpenVINO™ IR export of the VLM (not a PyTorch, safetensors, or ONNX checkpoint). A typical export contains:

Files

Purpose

openvino_language_model.{xml,bin}

Language-model decoder

openvino_text_embeddings_model.{xml,bin}

Text embeddings

openvino_vision_embeddings_model.{xml,bin}

Vision encoder

openvino_tokenizer.{xml,bin}, openvino_detokenizer.{xml,bin}

Tokenizer / detokenizer as IR

config.json, generation_config.json, preprocessor_config.json

Model, generation and preprocessing configuration

chat_template.jinja (or in tokenizer_config.json)

Prompt template

Exact file names vary slightly by architecture. Models published under the OpenVINO/ organization on Hugging Face are already in this format.

Get a different model#

Option A — make vlm-model with another model id (uses the same conversion script as the bundled model):

make vlm-model VLM_MODEL_ID=<org>/<model> VLM_DEVICE=CPU,GPU VLM_WEIGHT_FORMAT=int8

The target shows the model’s license and asks for confirmation first (ACCEPT_LICENSE=y skips the prompt). The result is resources/ov_models/<device>/<model>/ for each device listed. VLM_WEIGHT_FORMAT accepts int8 (default, smallest and fastest to load), int4 or fp16.

Option B — export with optimum-cli (any Hugging Face model id or local checkpoint that optimum-intel supports). Activate the export environment first, then:

optimum-cli export openvino \
  --model <org>/<model> --weight-format int8 \
  resources/ov_models/gpu/<model>

The export downloads the original weights from Hugging Face (several GB) and needs RAM roughly the size of the model. Add --trust-remote-code for models that ship their own modeling code (for example InternVL2). For a model that requires accepting a license on Hugging Face, run hf auth login first.

Repeat with resources/ov_models/cpu/<model> if a camera will run it on the CPU (the same export works on both devices; the folder is per device so a different precision can be used for each).

Option C — download a pre-exported model from the OpenVINO/ organization on Hugging Face. Activate the export environment first, then:

hf download OpenVINO/<model>-ov --local-dir resources/ov_models/gpu/<model>

Export environment#

Options B and C use their own Python environment, separate from the one make model creates. Create it once from uav-vision-analytics/ (requires python3-venv, see Troubleshooting):

python3 -m venv .cache/venv-vlm
source .cache/venv-vlm/bin/activate
pip install --upgrade pip
pip install "optimum[openvino]" huggingface_hub     # huggingface_hub alone is enough for Option C

In each new terminal, run source .cache/venv-vlm/bin/activate again before optimum-cli or hf. Run deactivate when done. The environment is under .cache/, which git ignores, and can be deleted afterwards.

Use Gemma 4#

Google’s Gemma 4 vision-language models work with the pipeline as-is; only the model folder and the captioning settings change. They need the OpenVINO™ GenAI runtime shipped in the DL Streamer Pipeline Server image.

  1. Set up the export environment, then download the pre-exported INT8 model (about 5 GB) for each device a camera will use:

    source .cache/venv-vlm/bin/activate
    hf download OpenVINO/gemma-4-E2B-it-int8-ov \
      --local-dir resources/ov_models/cpu/gemma-4-E2B-it-int8-ov
    hf download OpenVINO/gemma-4-E2B-it-int8-ov \
      --local-dir resources/ov_models/gpu/gemma-4-E2B-it-int8-ov
    

    To export it yourself instead (a different precision, for example), use Option B with --model google/gemma-4-E2B-it. The folder contains an extra openvino_text_embeddings_per_layer_model.{xml,bin} file that Gemma 4 needs; keep it.

  2. Select it in configs/cameras-pymavlink.yaml:

    captioning:
      enabled: true
      device: gpu
      model: gemma-4-E2B-it-int8-ov
      prompt: Describe the scene in one short sentence.
      max_new_tokens: 50
    
  3. Apply as described in Apply it.

Gemma 4 answers at length by default and may open with markdown headings or lists, so ask for brevity in the prompt. On the CPU, generating 50 tokens takes about 7 s, longer than the 5 s frame interval; use the GPU or lower max_new_tokens if captions fall behind.

Warning

Try Gemma 4 on one camera first. On an Intel Core Ultra iGPU it takes about 1 minute to load and about 17-27 s per caption. Loaded on the GPU next to two other GPU VLM instances and GPU detection, it failed with CL_OUT_OF_RESOURCES and left the pipeline server unresponsive (use docker restart uav-va-dlstreamer-pipeline-server to recover).

Apply it#

  1. Set captioning.model: <model> (and captioning.device) for the camera(s) in configs/cameras-pymavlink.yaml.

  2. Validate and apply:

    make check-cameras      # reports "captioning model not found" if the folder is missing
    make start-rtsp         # or disarm/arm; the file is re-read at every arming
    
  3. Watch the captions:

    docker exec uav-va-broker mosquitto_sub -t uav/caption/front_camera -C 2
    

For a pipeline started by hand, pass the container path in parameters.captioner_model_path (see Manage Pipelines — Start a pipeline).

Prompt and caption length#

prompt may contain letters, digits, spaces, and , . ? ' - only (validated by make check-cameras). max_new_tokens bounds the caption length and therefore the generation time; the pipeline sends one frame to the VLM every 5 s, so a model/prompt combination that takes longer than that per caption falls behind. Start around 50 tokens and raise it only if captions are cut short.

Troubleshooting#

captioning model not found, a captioning pipeline that goes to ERROR at start, and captions that arrive slowly or stop are covered in Troubleshooting — Models & Captioning. If the model loads but captions are poor, adjust the prompt first; fp16 weights on the GPU can be tried, INT8 rarely changes quality noticeably.