Troubleshooting#

Containers have started but the application is not working#

You can try resetting the volume storage by deleting the previously created volumes:

Note that this step does not apply when you are setting up the application for the first time.

source setup.sh --clean-data

OpenGL/Mesa Library Dependencies (Certain Kernel Versions)#

On some Linux systems with certain kernel versions, you may encounter OpenCV-related errors due to missing OpenGL/Mesa libraries. If you experience issues with the summary stack or video processing, try installing the following dependencies:

sudo apt update
sudo apt install libgl1-mesa-dri libgl1-mesa-dev

After installing these dependencies:

Remove the ov_models/ directory (if it exists)
Redeploy the VSS stack using the latest tagged images
Rerun your tests

This should resolve OpenCV-related dependency issues and allow the summary stack to work as expected.

Search returns no results after changing embedding model#

Problem: The UI displays No videos found matching your search query. Try using different keywords or check if videos have been uploaded. even though videos were ingested after running the setup script.

Cause: Either no videos have been processed yet, or the embedding model was switched to one with a different embedding dimension. Previously indexed vectors stay in the database, and their dimensions must match the active model. A mismatch prevents similarity lookups from returning any results.

Solution:

Verify at least one video has been uploaded or a summary run completed after the model change.
If you recently changed MULTIMODAL_EMBEDDING_MODEL or TEXT_EMBEDDING_MODEL, re-run ingestion so embeddings are recreated with the new dimensions. You can clean existing data with source setup.sh --clean-data and then bring the application back up with source setup.sh --search.
Review the supported embedding models and their dimensions in Supported Models for Multimodal Embedding Serving before switching models.

VLM Microservice Model Loading Issues#

Problem: VLM microservice fails to load or save models with permission errors, or you see errors related to model access in the logs.

Cause: This issue occurs when the ov-models Docker volume was created with incorrect ownership (root user) in previous versions of the application. The VLM microservice runs as a non-root user and requires proper permissions to read/write models.

Symptoms:

VLM microservice container fails to start or crashes during model loading
Permission denied errors in VLM service logs
Model conversion or caching failures
Error messages mentioning /home/appuser/.cache/huggingface or /app/ov-model access issues

Solution:

Stop the running application:
```
source setup.sh --down
```
Remove the existing ov-models (old volume name) and docker_ov-models (updated volume name) Docker volume:
```
docker volume rm ov-models docker_ov-models
```

Restart the application (the volume will be recreated with correct permissions):

# For Video Summarization
source setup.sh --summary

# Or for Video Search
source setup.sh --search

Note

Removing the ov-models or docker_ov-models volume will delete any previously cached or converted models. The VLM service will automatically re-download and convert models on the next startup, which may take additional time depending on your internet connection and the model size.

Prevention: This issue has been fixed in the current version of the VLM microservice Dockerfile. New installations will automatically create the volume with correct permissions.

VLM Final Summary Hallucination Issues#

Problem: The final summary generated by the VLM microservice contains hallucinated or inaccurate information that does not reflect the actual video content.

Cause: This issue can occur when using smaller VLM models that may not have sufficient capacity to accurately process and summarize complex video content, leading to generation of plausible but incorrect information.

Symptoms:

The final summary contains information not present in the video
The Summary describes events, objects, or activities that do not actually occur in the video
Inconsistent or contradictory information in the generated summary
The Summary quality is poor despite chunk-wise summaries being accurate

Solution: Try using a larger, more capable VLM model by updating the VLM_MODEL_NAME environment variable:

Stop the running application:
```
source setup.sh --down
```
Set a larger VLM model (e.g., upgrade from 4B to 8B parameters):
```
export VLM_MODEL_NAME="Qwen/Qwen3-VL-8B-Instruct"
```
Restart the application:
```
source setup.sh --summary
```

Alternative Models to Try:

For CPU: Qwen/Qwen3-VL-8B-Instruct (larger version)
For GPU: Consider other supported VLM models with higher parameter counts

Note

Larger models will require more system resources (RAM or VRAM) and may have longer inference times, but typically provide more accurate and coherent summaries.

Final Summary Stuck or OVMS Container Stopped#

Problem: The final video summary remains in a “Ready” or “In Progress” state indefinitely, and never completes.

Cause: The OVMS (OpenVINO™ Model Server) container may have crashed or the LLM request may have been rejected because the prompt size plus the requested max_completion_tokens exceeds the model’s maximum context length. For example, if a model supports a 4096-token context window and the application requests 4000 completion tokens, even a modest prompt (~300 tokens) will exceed the limit.

Symptoms:

Final summary status stays at “Ready” or “In Progress” and never progresses
OVMS container has exited (shows as stopped in docker ps -a)
OVMS logs contain errors like: Number of prompt tokens: <N> + max tokens value: <M> exceeds model max length: <L>
OVMS logs contain CL_OUT_OF_RESOURCES or similar GPU memory errors

Diagnosis:

Check if the OVMS container is still running:
```
docker ps -a | grep ovms
```
If the container has stopped or is in an exited state, check its logs:
```
docker logs <ovms-container-name> 2>&1 | tail -50
```
Look for errors related to token limits or resource exhaustion in the log output.

Solution:

If the logs show a token limit exceeded error, either reduce SUMMARIZATION_MAX_COMPLETION_TOKENS in your environment configuration, or switch to a model with a larger context window.
If the logs show GPU resource errors, see the section below on GPU memory issues.
After fixing the configuration, restart the application:
```
source setup.sh --down
source setup.sh --summary
```

Smaller Models May Block Final Summary Due to Limited Context Window#

Problem: The final video summary fails or hangs when using a smaller VLM/LLM model.

Cause: Smaller models often have a limited context window (e.g., 4096 tokens). When the combined prompt tokens and requested max_completion_tokens exceed this limit, the inference backend rejects the request and the final summary never completes.

Symptoms:

Final summary status stays at “Ready” or “In Progress” indefinitely
OVMS logs show errors such as: Number of prompt tokens: <N> + max tokens value: <M> exceeds model max length: <L>
The chunk-wise summaries complete successfully but the final summary does not

Solution:

Reduce PM_SUMMARIZATION_MAX_COMPLETION_TOKENS to a value below the default of 4000 so that the prompt plus completion tokens fit within the model’s context window:

export PM_SUMMARIZATION_MAX_COMPLETION_TOKENS=2000
source setup.sh --summary

Alternatively, switch to a model with a larger context window.

VLM Workload Fails on NPU#

Problem: The VLM model fails to load or run when VLM_TARGET_DEVICE or LLM_TARGET_DEVICE is set to NPU.

Cause: Not all VLM/LLM models are compatible with NPU execution. NPU support depends on the model architecture and the OpenVINO™ version installed.

Symptoms:

OVMS container crashes or fails to start when targeting NPU
Inference errors or unsupported-operation messages in OVMS logs
Model conversion succeeds but inference produces errors

Solution:

Verify that your model is listed on the OpenVINO™ Supported Models page for NPU execution.
If the model is not supported on NPU, switch to a supported model or fall back to CPU/GPU:
```
export VLM_TARGET_DEVICE="CPU"
source setup.sh --summary
```

vLLM XPU Fails to Start with “No available memory for the cache blocks”#

Problem: With ENABLE_VLLM_GPU=true, the vllm-xpu-service container never becomes healthy and restarts in a loop. The logs end with:

INFO [gpu_worker.py:640] Available KV cache memory: -1.66 GiB
ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` ...

A related variant appears once the KV cache is positive but still too small:

ValueError: To serve at least one request with the model's max seq len (32000),
(4.39 GiB KV cache is needed, which is larger than the available KV cache memory (0.93 GiB)

Because pipeline-manager waits on vllm-xpu-service being healthy, pipeline-manager, nginx, and the UI stay in Created state and the application never comes up.

Cause: The GPU does not have enough memory for all three consumers at once — model weights, vision-encoder activations, and KV cache. vLLM reserves total_memory x VLLM_GPU_MEM, subtracts weights and activations, and uses whatever is left for KV cache. Qwen/Qwen3-VL-4B-Instruct in bfloat16 needs about 8.6 GB for weights alone, and the vision encoder profiles against a full-size image (16384 encoder tokens by default), which costs several GB more. This fits on a 24 GB Arc Pro B60 but not on a 12 GB card such as the Arc B580.

Solution: Check how much memory your GPU actually reports, then reduce one or more of the three consumers.

Confirm the device and its total memory:

docker run --rm --device /dev/dri:/dev/dri --group-add "$(getent group render | cut -d: -f3)" \
  --entrypoint python3 vllm/vllm-openai-xpu:v0.31.0 -c \
  "import torch; p = torch.xpu.get_device_properties(0); print(p.name, round(p.total_memory/1024**3, 2), 'GiB')"

Apply the fixes below, cheapest first, then restart with source setup.sh --down followed by your usual setup.sh command.
- Use the FP8 checkpoint — the single biggest win. This halves the weight footprint from 8.6 GB to 5.9 GB with no change to context length:
```
export VLM_MODEL_NAME="Qwen/Qwen3-VL-4B-Instruct-FP8"
```
- Cap the image size — the vision encoder, not the weights, is usually the largest remaining consumer. VLLM_MM_MAX_PIXELS is expressed in pixels and defaults to 16777216:
```
export VLLM_MM_MAX_PIXELS=401408
```
  How to choose a value. vLLM sizes the encoder on the worst-case frame the cap allows, not on your actual frames:
```
worst_case_tokens = VLLM_MM_MAX_PIXELS / 1024
encoder_budget    = max(VLLM_MAX_NUM_BATCHED_TOKENS, worst_case_tokens)
```
  The 1024 is (patch_size x merge_size)^2, which is (16 x 2)^2 for Qwen3-VL. Encoder memory scales with encoder_budget, so the cap is the main lever — but only down to the VLLM_MAX_NUM_BATCHED_TOKENS floor (2048 by default). Below that it frees no further memory and only downscales frames more.
  
  VLLM_MM_MAX_PIXELS
  
  Worst-case tokens
  
  A 1920x1080 frame becomes
  
  16777216 (default)
  
  16384
  
  1920x1088, unchanged
  
  401408
  
  392
  
  832x448
  
  262144
  
  256
  
  672x384
  
  The cap is a pixel-area budget, so the resulting dimensions follow the aspect ratio of the source frame. It limits the resolution of each frame, not how many frames you send — the default PM_MULTI_FRAME_COUNT=12 works unchanged. Lowering it trades caption detail for memory, so raise it again if captions become too vague.
- Lower the context length — the error message tells you exactly how much KV cache one request at VLLM_MAX_MODEL_LEN requires. Scale the value down by the ratio of available to needed memory:
```
export VLLM_MAX_MODEL_LEN=24000
```
- Raise the memory fraction — only a small gain, and setting it too close to 1.0 risks out-of-memory errors during inference rather than at startup:
```
export VLLM_GPU_MEM=0.95
```

Working configurations on a 12 GB Intel Arc B580, using vllm/vllm-openai-xpu:v0.31.0:

Model	`VLLM_GPU_MEM`	`VLLM_MAX_MODEL_LEN`	`VLLM_MM_MAX_PIXELS`	Usable context
`Qwen3-VL-4B-Instruct-FP8`	`0.95`	`24000`	`401408`	25,659 tokens
`Qwen3-VL-4B-Instruct` (bf16)	`0.95`	`8192`	`401408`	10,496 tokens

The first row is the recommended recipe, and all four of its settings are required: weakening any single one — leaving the memory fraction at 0.9, dropping the pixel cap, keeping 32000 context, or reverting to bfloat16 — still fails to start on this card. The second row shows bfloat16 remains usable if you accept a much shorter context instead of switching to FP8.

Note

Only vLLM on Intel Arc GPUs is affected. The CPU vLLM backend and the default OVMS backend size their caches differently and are unaffected by these variables.

Embedding Fails on NPU/GPU in Kubernetes (Device Permission)#

Problem: On a Helm deployment, video embedding fails when multimodal-embedding-ms or multimodal-dataprep is configured for NPU (or GPU), even though the device plugin granted the resource and /dev/accel (or /dev/dri) is mounted in the pod. The UI may show a generic embedding failure.

Cause: The container runs as a non-root user. The accelerator device node (/dev/accel for NPU, /dev/dri for GPU) is owned by a host group (gid). Without that gid in the pod’s supplementalGroups, the user cannot open the device, so OpenVINO™ enumerates only ['CPU'] and NPU/GPU plugin initialization fails — sometimes with a misleading “cannot load library …compiler_loader.so” message even though the library exists.

Symptoms:

multimodal-embedding-ms / multimodal-dataprep logs show DEVICES: ['CPU'] (NPU/GPU missing) and an NPU/GPU plugin/compiler load failure.
The pod has the device mounted and the resource granted, yet inference stays on CPU or errors out.

Solution:

Find the gid that owns the device node on the scheduled worker node:
```
ls -ln /dev/accel   # NPU
ls -ln /dev/dri     # GPU
```
Set global.accelGroupIds in user_values_override.yaml to match (it is injected into the pod supplementalGroups when a service uses GPU/NPU):
```
global:
  accelGroupIds:
    - 992   # replace with the gid owning /dev/accel or /dev/dri on your node
```
Re-deploy (or helm upgrade) and confirm the pod now lists ['CPU','NPU'] (or GPU) and loads the model on the accelerator.

Note

Nodes are heterogeneous, so the gid can differ per node. The chart cannot auto-detect it (templates render client-side), which is why global.accelGroupIds is an explicit override. On first GPU/NPU start the model is compiled and cached to ovCacheDir (/app/ov_models/ov_cache); allow the DataPrep startupProbe budget to complete this one-time compile.

OVMS KV Cache Exhaustion#

Problem: LLM or VLM inference requests are slow, produce incomplete responses, or fail under concurrent usage.

Cause: The OVMS KV cache is fully consumed. The KV cache holds intermediate attention state during text generation; when it is exhausted, OVMS must preempt or reject new requests.

Symptoms:

OVMS logs show cache usage at or near 100 %:

llm_executor.hpp:104] All requests: 1; Scheduled requests: 1; Cache type: static, cache usage: 100.0% of 4.0 GB;

Requests take significantly longer than expected or time out
Under concurrent requests, some are rejected or produce truncated output

Diagnosis:

Check OVMS container logs for cache usage lines:

docker logs <ovms-container-name> 2>&1 | grep "cache usage"

If usage is consistently at or near 100 %, the cache is too small for your workload.

Solution:

Increase the KV cache size by setting the OVMS_CACHE_SIZE_GB environment variable before running the setup script. The default is dynamically calculated based on available memory:

Device	Allocation	Clamp range
CPU	25 % of system RAM	[2, 16] GB
Integrated GPU (iGPU)	25 % of system RAM	[2, 6] GB
Discrete GPU (dGPU)	33 % of dedicated VRAM	[2, 16] GB

The setup script automatically detects the GPU type using the OpenVINO™ runtime. Setting OVMS_CACHE_SIZE_GB overrides the dynamic calculation for all device types.

# Example: set KV cache to 8 GB
export OVMS_CACHE_SIZE_GB=8
source setup.sh --summary   # or --search

The updated cache size is applied to the existing model configuration on the next run — no re-export is required.

Note

On integrated GPUs (iGPU), memory is shared with the system. Setting a very large cache size may leave insufficient memory for model weights and cause CL_OUT_OF_RESOURCES errors. Start with modest increases (e.g., 4 → 6 → 8 GB) and monitor both cache usage and GPU memory utilization.

4K/8K Video Ingestion Stalls with a Worker Timeout#

Problem: Ingesting a high-resolution (4K/8K) video in --search, --dual, or --unified mode never completes. Search never returns results for that video, and the multimodal-dataprep container reboots its worker mid-ingestion.

Cause: The DataPrep decoder moves each frame between pipeline stages through a pool of fixed-size shared-memory blocks. Each decoded frame is written into a single block as a raw RGB buffer of exactly width × height × 3 bytes. The block size is SDK_VIDEO_SHM_BLOCK_SIZE, which defaults to 6220800 = 1920 × 1080 × 3 (1080p). A 4K or 8K frame is several times larger than the default block, so the write fails, the frame is never enqueued, and every downstream stage (detection → embed → store → result) starves until the worker hits the Gunicorn timeout and is force-killed.

Symptoms:

Every pipeline stage logs an empty queue, then the worker aborts with signal 134 (SIGABRT) and leaks shared-memory objects. Inspect with docker logs multimodal-dataprep:

WARNING: | detection_worker | [DETECTION QUEUE EMPTY] WAITING...
WARNING: | store_worker      | [STORE_WORKER] Queue empty, waiting...
WARNING: | embed_worker      | [EMBED_WORKER] Queue empty, waiting...
WARNING: | process_result_worker | [RESULT WORKER] Queue empty, waiting...
[CRITICAL] WORKER TIMEOUT (pid:8)
[ERROR] Worker (pid:8) was sent code 134!
UserWarning: resource_tracker: There appear to be 1024 leaked shared_memory objects to clean up at shutdown

Only lower-resolution (1080p or smaller) videos index successfully.

Solution:

Set SDK_VIDEO_SHM_BLOCK_SIZE to at least width × height × 3 for your highest-resolution source. Use the table below:

Source resolution	Pixels (W × H)	Minimum `SDK_VIDEO_SHM_BLOCK_SIZE` (`W × H × 3`)
1080p (default)	1920 × 1080	`6220800`
4K UHD	3840 × 2160	`24883200`
DCI 4K	4096 × 2160	`26542080`
8K UHD	7680 × 4320	`99532800`

Export the value (or set it in your .env) before re-running your deployment mode:

# Example: enable 4K ingestion (3840 x 2160 x 3 = 24883200 bytes per block)
export SDK_VIDEO_SHM_BLOCK_SIZE=24883200
source setup.sh --search   # re-run your mode so the new value applies

Budget the total shared memory. The pool pre-allocates SDK_VIDEO_SHM_MAX_BLOCKS × SDK_VIDEO_SHM_BLOCK_SIZE bytes in the host /dev/shm. With the default 512 blocks this is ≈ 12.7 GB for 4K and ≈ 51 GB for 8K. If the host cannot spare that much (check with df -h /dev/shm), lower SDK_VIDEO_SHM_MAX_BLOCKS so the product fits:
```
export SDK_VIDEO_SHM_BLOCK_SIZE=24883200   # 4K frame size
export SDK_VIDEO_SHM_MAX_BLOCKS=128        # 128 x 24883200 ≈ 3.2 GB of /dev/shm
source setup.sh --search
```

Note

Always size the block from the largest resolution you will ingest — an oversized block only wastes memory, while an undersized one triggers the failure above. For the full explanation, see the DataPrep Get Started guide.

Accuracy of search results#

The accuracy of search results vary based on the embedding model used, configuration on frame sampling, object detection enabled or disabled, and the diversity of the video contents. The user is encouraged to check on these aspects in case the accuracy of the search results is not found to be satisfactory. Note that higher accuracy is normally a tradeoff with performance. Some specific pointers are provided below:

Model selection: Among the supported models, models with higher dimensionality will provide better results.
Higher frame sampling leads to better accuracy but at the cost of higher compute requirements.
Enabling object detection normally provides a better accuracy. Consider this option in alignment with selected model capability.
If the video diversity is very low, any query will seem to return the same results. Example: Same camera feed or video used for testing will return results from the same video irrespective of the query. Check the relevance score to determine how strong the match is.

Raise an issue in case of continued challenges faced.

`VLLM_MM_MAX_PIXELS`	Worst-case tokens	A 1920x1080 frame becomes
`16777216` (default)	16384	1920x1088, unchanged
`401408`	392	832x448
`262144`	256	672x384

Troubleshooting#

Containers have started but the application is not working#

OpenGL/Mesa Library Dependencies (Certain Kernel Versions)#

Search returns no results after changing embedding model#

VLM Microservice Model Loading Issues#

VLM Final Summary Hallucination Issues#

Final Summary Stuck or OVMS Container Stopped#

Smaller Models May Block Final Summary Due to Limited Context Window#

VLM Workload Fails on NPU#

GPU Out-of-Resources When Loading Multiple Models#

vLLM XPU Fails to Start with “No available memory for the cache blocks”#

Embedding Fails on NPU/GPU in Kubernetes (Device Permission)#

OVMS KV Cache Exhaustion#

4K/8K Video Ingestion Stalls with a Worker Timeout#

Accuracy of search results#