Troubleshooting#
Containers have started but the application is not working#
You can try resetting the volume storage by deleting the previously created volumes:
Note that this step does not apply when you are setting up the application for the first time.
source setup.sh --clean-data
OpenGL/Mesa Library Dependencies (Certain Kernel Versions)#
On some Linux systems with certain kernel versions, you may encounter OpenCV-related errors due to missing OpenGL/Mesa libraries. If you experience issues with the summary stack or video processing, try installing the following dependencies:
sudo apt update
sudo apt install libgl1-mesa-dri libgl1-mesa-dev
After installing these dependencies:
Remove the
ov_models/directory (if it exists)Redeploy the VSS stack using the latest tagged images
Rerun your tests
This should resolve OpenCV-related dependency issues and allow the summary stack to work as expected.
Search returns no results after changing embedding model#
Problem: The UI displays No videos found matching your search query. Try using different keywords or check if videos have been uploaded. even though videos were ingested after running the setup script.
Cause: Either no videos have been processed yet, or the embedding model was switched to one with a different embedding dimension. Previously indexed vectors stay in the database, and their dimensions must match the active model. A mismatch prevents similarity lookups from returning any results.
Solution:
Verify at least one video has been uploaded or a summary run completed after the model change.
If you recently changed
MULTIMODAL_EMBEDDING_MODELorTEXT_EMBEDDING_MODEL, re-run ingestion so embeddings are recreated with the new dimensions. You can clean existing data withsource setup.sh --clean-dataand then bring the application back up withsource setup.sh --search.Review the supported embedding models and their dimensions in Supported Models for Multimodal Embedding Serving before switching models.
Model-download cache permission errors#
Problem: Setup stops before the model-download container becomes healthy, with Failed to initialize cache and Permission denied for /opt/models/.model-download-cache/uv.
Cause: The container runs as a non-root user whose UID can differ from the host user. Adding the host group to the container does not grant write access to a host directory that is not group-writable.
Solution: Use the updated setup.sh. It places the UV dependency cache in the container’s /tmp/model-download-cache/uv, which is discarded when the transient container is removed. Downloaded models remain persistent in the host’s ov_models/ directory.
Before starting model-download, setup assigns ov_models/ to the host user’s primary group and grants that group read, write, and traversal access. It stops with an error if this preparation fails. For an existing model tree owned by another user, have its owner or administrator grant the intended shared group access to the affected directories and files, then retry. Do not run model-download as root or make the model directory world-writable.
VLM Microservice Model Loading Issues#
Problem: VLM microservice fails to load or save models with permission errors, or you see errors related to model access in the logs.
Cause: This issue occurs when the ov-models Docker volume was created with incorrect ownership (root user) in previous versions of the application. The VLM microservice runs as a non-root user and requires proper permissions to read/write models.
Symptoms:
VLM microservice container fails to start or crashes during model loading
Permission denied errors in VLM service logs
Model conversion or caching failures
Error messages mentioning
/home/appuser/.cache/huggingfaceor/app/ov-modelaccess issues
Solution:
Stop the running application:
source setup.sh --down
Remove the existing
ov-models(old volume name) anddocker_ov-models(updated volume name) Docker volume:docker volume rm ov-models docker_ov-models
Restart the application (the volume will be recreated with correct permissions):
# For Video Summarization source setup.sh --summary # Or for Video Search source setup.sh --search
Note
Removing the ov-models or docker_ov-models volume will delete any previously cached or converted models. The VLM service will automatically re-download and convert models on the next startup, which may take additional time depending on your internet connection and the model size.
Prevention: This issue has been fixed in the current version of the VLM microservice Dockerfile. New installations will automatically create the volume with correct permissions.
VLM Final Summary Hallucination Issues#
Problem: The final summary generated by the VLM microservice contains hallucinated or inaccurate information that does not reflect the actual video content.
Cause: This issue can occur when using smaller VLM models that may not have sufficient capacity to accurately process and summarize complex video content, leading to generation of plausible but incorrect information.
Symptoms:
The final summary contains information not present in the video
The Summary describes events, objects, or activities that do not actually occur in the video
Inconsistent or contradictory information in the generated summary
The Summary quality is poor despite chunk-wise summaries being accurate
Solution:
Try using a larger, more capable VLM model by updating the VLM_MODEL_NAME environment variable:
Stop the running application:
source setup.sh --down
Set a larger VLM model (e.g., upgrade from 4B to 9B parameters):
export VLM_MODEL_NAME="OpenVINO/Qwen3.5-9B-int4-ov"
Restart the application:
source setup.sh --summary
Alternative Models to Try:
For OVMS on CPU or GPU:
OpenVINO/Qwen3.5-9B-int4-ov(larger version) orOpenVINO/gemma-4-E4B-it-int8-ovFor vLLM:
Qwen/Qwen3-VL-8B-Instruct(larger version)
Note
Larger models will require more system resources (RAM or VRAM) and may have longer inference times, but typically provide more accurate and coherent summaries.
Final Summary Stuck or OVMS Container Stopped#
Problem: The final video summary remains in a “Ready” or “In Progress” state indefinitely, and never completes.
Cause: The OVMS (OpenVINO™ Model Server) container may have crashed or the LLM request may have been rejected because the prompt size plus the requested max_completion_tokens exceeds the model’s maximum context length. For example, if a model supports a 4096-token context window and the application requests 4000 completion tokens, even a modest prompt (~300 tokens) will exceed the limit.
Symptoms:
Final summary status stays at “Ready” or “In Progress” and never progresses
OVMS container has exited (shows as stopped in
docker ps -a)OVMS logs contain errors like:
Number of prompt tokens: <N> + max tokens value: <M> exceeds model max length: <L>OVMS logs contain
CL_OUT_OF_RESOURCESor similar GPU memory errors
Diagnosis:
Check if the OVMS container is still running:
docker ps -a | grep ovms
If the container has stopped or is in an exited state, check its logs:
docker logs <ovms-container-name> 2>&1 | tail -50
Look for errors related to token limits or resource exhaustion in the log output.
Solution:
If the logs show a token limit exceeded error, either reduce
SUMMARIZATION_MAX_COMPLETION_TOKENSin your environment configuration, or switch to a model with a larger context window.If the logs show GPU resource errors, see the section below on GPU memory issues.
After fixing the configuration, restart the application:
source setup.sh --down source setup.sh --summary
Smaller Models May Block Final Summary Due to Limited Context Window#
Problem: The final video summary fails or hangs when using a smaller VLM/LLM model.
Cause: Smaller models often have a limited context window (e.g., 4096 tokens). When the combined prompt tokens and requested max_completion_tokens exceed this limit, the inference backend rejects the request and the final summary never completes.
Symptoms:
Final summary status stays at “Ready” or “In Progress” indefinitely
OVMS logs show errors such as:
Number of prompt tokens: <N> + max tokens value: <M> exceeds model max length: <L>The chunk-wise summaries complete successfully but the final summary does not
Solution:
Reduce PM_SUMMARIZATION_MAX_COMPLETION_TOKENS to a value below the default of 4000 so that the prompt plus completion tokens fit within the model’s context window:
export PM_SUMMARIZATION_MAX_COMPLETION_TOKENS=2000
source setup.sh --summary
Alternatively, switch to a model with a larger context window.
VLM Workload Fails on NPU#
Problem: Chunk captioning fails, or OVMS fails to load or run the model, when VLM_TARGET_DEVICE is set to NPU.
Cause: The VLM is not supported on NPU for VSS. NPU vision-language pipelines accept only one image per request, while VSS sends several frames per chunk. Only the final-summary LLM can run on NPU.
Symptoms:
OVMS container crashes or fails to start when targeting NPU
Inference errors or unsupported-operation messages in OVMS logs
Chunk captioning requests fail while the model is loaded
Solution:
Run the VLM on CPU or GPU:
export VLM_TARGET_DEVICE="GPU" source setup.sh --summary
To use the NPU, move only the final-summary LLM to it with a text-only
-cw-model. See Final Summary Fails or Is Cut Short with the LLM on NPU.
Final Summary Fails or Is Cut Short with the LLM on NPU#
Problem: With LLM_TARGET_DEVICE=NPU, setup stops with LLM_TARGET_DEVICE=NPU needs a text-only LLM, the final or audio summary fails, or the summary ends mid-sentence.
Cause: An NPU LLM is compiled for a fixed prompt length (OVMS_NPU_MAX_PROMPT_LEN, default 8192 tokens) and response length (OVMS_NPU_MIN_RESPONSE_LEN, default 1024). OVMS rejects longer prompts with Input length exceeds the maximum allowed length and cuts longer output without an error. Vision-language models on NPU cannot have their prompt limit raised, so they are rejected for the LLM role.
Solution:
Use a text-only LLM on NPU, for example
OVMS_LLM_MODEL_NAME=OpenVINO/Phi-3.5-mini-instruct-int4-cw-ovorOpenVINO/Qwen3-8B-int4-cw-ov, or move the LLM to GPU/CPU.Leave
PM_LLM_MAX_CONTEXT_LENGTHandPM_SUMMARIZATION_MAX_COMPLETION_TOKENSunset so the setup script sizes them to the NPU limits, or lower them below the values printed in theNPU LLM limitslog line.For long, speech-heavy videos, raise the limit (for example
export OVMS_NPU_MAX_PROMPT_LEN=16384, which needs about 20 GB of system memory forPhi-3.5-mini-instruct) or disable audio. On systems with 16 GB of memory or less, lower it to4096if OVMS fails to load the model on NPU. A single audio transcript longer than the prompt budget is sent in one request and fails on NPU.
On Helm, set ovms.env.OVMS_NPU_MAX_PROMPT_LEN and ovms.env.OVMS_NPU_MIN_RESPONSE_LEN, then restart the OVMS pod.
OpenVINO Model Download Fails with Failed to download llm model#
Problem: Setup stops with cannot be downloaded for NPU or is an NPU-optimized channel-wise repo, or the model download fails with ValueError: ('Failed to download llm model', 'OpenVINO/...').
Cause: The model-download service pulls a pre-converted OpenVINO/* repo only when the repo matches the target device. NPU needs an NPU-optimized channel-wise repo whose name contains -cw-. CPU and GPU need a repo without -cw-. On a mismatch, the service tries to convert the repo instead, which fails for OpenVINO/* repos.
Solution:
On NPU, use a
-cw-repo, for exampleOpenVINO/Phi-3.5-mini-instruct-int4-cw-ovinstead ofOpenVINO/Phi-3.5-mini-instruct-int4-gq-ov.On CPU or GPU, use the variant without
-cw-, for exampleOpenVINO/Phi-3.5-mini-instruct-int4-ov.
GPU Out-of-Resources When Loading Multiple Models#
Problem: OVMS crashes or fails inference when multiple models (e.g., VLM + LLM) are loaded on the same GPU.
Cause: Loading multiple large models on a single GPU can exceed the available device memory. When the GPU runs out of resources during inference, the OpenCL runtime returns CL_OUT_OF_RESOURCES and OVMS terminates the request or crashes.
Symptoms:
OVMS container exits unexpectedly or restarts repeatedly
OVMS logs contain errors like:
onednn_verbose,v1,primitive,error,ocl,errcode -5,CL_OUT_OF_RESOURCES Exception from src/plugins/intel_gpu/src/graph/impls/onednn/primitive_onednn_base.h Error occurred in LLM executor
Inference requests hang and then fail
Only one model works at a time but loading both causes failures
Solution:
Distribute models across devices — run the VLM on GPU and the LLM on CPU (or vice versa) to avoid competing for GPU memory. Adjust the device settings in your environment configuration accordingly.
Use smaller model variants — switch to quantized or smaller parameter models that consume less GPU memory.
Increase GPU resources — if available, use a GPU with more memory.
After making changes, restart the application:
source setup.sh --down source setup.sh --summary
vLLM XPU Fails to Start with “No available memory for the cache blocks”#
Problem: With ENABLE_VLLM_GPU=true, the vllm-xpu-service container never becomes healthy and restarts in a loop. The logs end with:
INFO [gpu_worker.py:640] Available KV cache memory: -1.66 GiB
ValueError: No available memory for the cache blocks. Try increasing `gpu_memory_utilization` ...
A related variant appears once the KV cache is positive but still too small:
ValueError: To serve at least one request with the model's max seq len (32000),
(4.39 GiB KV cache is needed, which is larger than the available KV cache memory (0.93 GiB)
Because pipeline-manager waits on vllm-xpu-service being healthy, pipeline-manager, nginx, and the UI stay in Created state and the application never comes up.
Cause: The GPU does not have enough memory for all three consumers at once — model weights, vision-encoder activations, and KV cache. vLLM reserves total_memory x VLLM_GPU_MEM, subtracts weights and activations, and uses whatever is left for KV cache. Qwen/Qwen3-VL-4B-Instruct in bfloat16 needs about 8.6 GB for weights alone, and the vision encoder profiles against a full-size image (16384 encoder tokens by default), which costs several GB more. This fits on a 24 GB Arc Pro B60 but not on a 12 GB card such as the Arc B580.
Solution: Check how much memory your GPU actually reports, then reduce one or more of the three consumers.
Confirm the device and its total memory:
docker run --rm --device /dev/dri:/dev/dri --group-add "$(getent group render | cut -d: -f3)" \ --entrypoint python3 vllm/vllm-openai-xpu:v0.31.0 -c \ "import torch; p = torch.xpu.get_device_properties(0); print(p.name, round(p.total_memory/1024**3, 2), 'GiB')"
Apply the fixes below, cheapest first, then restart with
source setup.sh --downfollowed by your usualsetup.shcommand.Use the FP8 checkpoint — the single biggest win. This halves the weight footprint from 8.6 GB to 5.9 GB with no change to context length:
export VLM_MODEL_NAME="Qwen/Qwen3-VL-4B-Instruct-FP8"
Cap the image size — the vision encoder, not the weights, is usually the largest remaining consumer.
VLLM_MM_MAX_PIXELSis expressed in pixels and defaults to16777216:export VLLM_MM_MAX_PIXELS=401408
How to choose a value. vLLM sizes the encoder on the worst-case frame the cap allows, not on your actual frames:
worst_case_tokens = VLLM_MM_MAX_PIXELS / 1024 encoder_budget = max(VLLM_MAX_NUM_BATCHED_TOKENS, worst_case_tokens)
The
1024is(patch_size x merge_size)^2, which is(16 x 2)^2for Qwen3-VL. Encoder memory scales withencoder_budget, so the cap is the main lever — but only down to theVLLM_MAX_NUM_BATCHED_TOKENSfloor (2048by default). Below that it frees no further memory and only downscales frames more.VLLM_MM_MAX_PIXELSWorst-case tokens
A 1920x1080 frame becomes
16777216(default)16384
1920x1088, unchanged
401408392
832x448
262144256
672x384
The cap is a pixel-area budget, so the resulting dimensions follow the aspect ratio of the source frame. It limits the resolution of each frame, not how many frames you send — the default
PM_MULTI_FRAME_COUNT=12works unchanged. Lowering it trades caption detail for memory, so raise it again if captions become too vague.Lower the context length — the error message tells you exactly how much KV cache one request at
VLLM_MAX_MODEL_LENrequires. Scale the value down by the ratio of available to needed memory:export VLLM_MAX_MODEL_LEN=24000
Raise the memory fraction — only a small gain, and setting it too close to
1.0risks out-of-memory errors during inference rather than at startup:export VLLM_GPU_MEM=0.95
Working configurations on a 12 GB Intel Arc B580, using vllm/vllm-openai-xpu:v0.31.0:
Model |
|
|
|
Usable context |
|---|---|---|---|---|
|
|
|
|
25,659 tokens |
|
|
|
|
10,496 tokens |
The first row is the recommended recipe, and all four of its settings are required: weakening any single one — leaving the memory fraction at 0.9, dropping the pixel cap, keeping 32000 context, or reverting to bfloat16 — still fails to start on this card. The second row shows bfloat16 remains usable if you accept a much shorter context instead of switching to FP8.
Note
Only vLLM on Intel Arc GPUs is affected. The CPU vLLM backend and the default OVMS backend size their caches differently and are unaffected by these variables.
Embedding Fails on NPU/GPU in Kubernetes (Device Permission)#
Problem: On a Helm deployment, video embedding fails when multimodal-embedding-ms or multimodal-dataprep is configured for NPU (or GPU), even though the device plugin granted the resource and /dev/accel (or /dev/dri) is mounted in the pod. The UI may show a generic embedding failure.
Cause: The container runs as a non-root user. The accelerator device node (/dev/accel for NPU, /dev/dri for GPU) is owned by a host group (gid). Without that gid in the pod’s supplementalGroups, the user cannot open the device, so OpenVINO™ enumerates only ['CPU'] and NPU/GPU plugin initialization fails — sometimes with a misleading “cannot load library …compiler_loader.so” message even though the library exists.
Symptoms:
multimodal-embedding-ms/multimodal-datapreplogs showDEVICES: ['CPU'](NPU/GPU missing) and an NPU/GPU plugin/compiler load failure.The pod has the device mounted and the resource granted, yet inference stays on CPU or errors out.
Solution:
Find the gid that owns the device node on the scheduled worker node:
ls -ln /dev/accel # NPU ls -ln /dev/dri # GPU
Set
global.accelGroupIdsinuser_values_override.yamlto match (it is injected into the podsupplementalGroupswhen a service uses GPU/NPU):global: accelGroupIds: - 992 # replace with the gid owning /dev/accel or /dev/dri on your node
Re-deploy (or
helm upgrade) and confirm the pod now lists['CPU','NPU'](orGPU) and loads the model on the accelerator.
Note
Nodes are heterogeneous, so the gid can differ per node. The chart cannot auto-detect it (templates render client-side), which is why global.accelGroupIds is an explicit override. On first GPU/NPU start the model is compiled and cached to ovCacheDir (/app/ov_models/ov_cache); allow the DataPrep startupProbe budget to complete this one-time compile.
OVMS KV Cache Exhaustion#
Problem: LLM or VLM inference requests are slow, produce incomplete responses, or fail under concurrent usage.
Cause: The OVMS KV cache is fully consumed. The KV cache holds intermediate attention state during text generation; when it is exhausted, OVMS must preempt or reject new requests.
Symptoms:
OVMS logs show cache usage at or near 100 %:
llm_executor.hpp:104] All requests: 1; Scheduled requests: 1; Cache type: static, cache usage: 100.0% of 4.0 GB;
Requests take significantly longer than expected or time out
Under concurrent requests, some are rejected or produce truncated output
Diagnosis:
Check OVMS container logs for cache usage lines:
docker logs <ovms-container-name> 2>&1 | grep "cache usage"
If usage is consistently at or near 100 %, the cache is too small for your workload.
Solution:
Increase the KV cache size by setting the OVMS_CACHE_SIZE_GB environment variable before running the setup script. The default is dynamically calculated based on available memory:
Device |
Allocation |
Clamp range |
|---|---|---|
CPU |
25 % of system RAM |
[2, 16] GB |
Integrated GPU (iGPU) |
25 % of system RAM |
[2, 6] GB |
Discrete GPU (dGPU) |
33 % of dedicated VRAM |
[2, 16] GB |
The setup script detects a discrete GPU by reading its dedicated VRAM from sysfs; if none is found, an Intel GPU at PCI address 0000:00:02.0 is treated as an iGPU. Setting OVMS_CACHE_SIZE_GB overrides the dynamic calculation for all device types.
# Example: set KV cache to 8 GB
export OVMS_CACHE_SIZE_GB=8
source setup.sh --summary # or --search
The updated cache size is applied to the existing model configuration on the next run — no re-export is required.
Note
On integrated GPUs (iGPU), memory is shared with the system. Setting a very large cache size may leave insufficient memory for model weights and cause CL_OUT_OF_RESOURCES errors. Start with modest increases (e.g., 4 → 6 → 8 GB) and monitor both cache usage and GPU memory utilization.
OVMS Fails to Start with reserved_la_bytes <= budget_in_bytes#
Problem: setup.sh stops with dependency failed to start: container ... ovms-service ... exited (sometimes exit code 139), or the OVMS logs show:
Error during llm node initialization ... Check 'reserved_la_bytes <= budget_in_bytes' failed
Cause: Hybrid linear-attention models such as Qwen3.5 reserve a fixed recurrent state for every sequence OVMS may batch (max_num_seqs in graph.pbtxt), and this reservation comes out of the KV cache budget (cache_size). For example, Qwen3.5-4B needs about 25 MB per sequence, so max_num_seqs: 256 (about 6.3 GB) does not fit the 6 GB integrated-GPU cache.
Solution: setup.sh sets max_num_seqs to the larger of 16 and PM_VLM_CONCURRENT + PM_LLM_CONCURRENT. Pipeline Manager never sends more concurrent requests than that, and any extra requests queue. The value is written to newly downloaded models and patched into existing graph.pbtxt files on the next run, so no re-export is required. To override it:
export OVMS_MAX_NUM_SEQS=8
source setup.sh --summary
Alternatively, increase OVMS_CACHE_SIZE_GB if the device has memory to spare.
On Helm, the OVMS download init containers apply ovms.env.OVMS_MAX_NUM_SEQS (default 16) and ovms.env.OVMS_CACHE_SIZE_GB the same way. Restart the OVMS pod after changing them.
4K/8K Video Ingestion Stalls with a Worker Timeout#
Problem: Ingesting a high-resolution (4K/8K) video in --search, --dual, or --unified mode never completes. Search never returns results for that video, and the multimodal-dataprep container reboots its worker mid-ingestion.
Cause: The DataPrep decoder moves each frame between pipeline stages through a pool of fixed-size shared-memory blocks. Each decoded frame is written into a single block as a raw RGB buffer of exactly width × height × 3 bytes. The block size is SDK_VIDEO_SHM_BLOCK_SIZE, which defaults to 6220800 = 1920 × 1080 × 3 (1080p). A 4K or 8K frame is several times larger than the default block, so the write fails, the frame is never enqueued, and every downstream stage (detection → embed → store → result) starves until the worker hits the Gunicorn timeout and is force-killed.
Symptoms:
Every pipeline stage logs an empty queue, then the worker aborts with signal
134(SIGABRT) and leaks shared-memory objects. Inspect withdocker logs multimodal-dataprep:WARNING: | detection_worker | [DETECTION QUEUE EMPTY] WAITING... WARNING: | store_worker | [STORE_WORKER] Queue empty, waiting... WARNING: | embed_worker | [EMBED_WORKER] Queue empty, waiting... WARNING: | process_result_worker | [RESULT WORKER] Queue empty, waiting... [CRITICAL] WORKER TIMEOUT (pid:8) [ERROR] Worker (pid:8) was sent code 134! UserWarning: resource_tracker: There appear to be 1024 leaked shared_memory objects to clean up at shutdown
Only lower-resolution (1080p or smaller) videos index successfully.
Solution:
Set
SDK_VIDEO_SHM_BLOCK_SIZEto at leastwidth × height × 3for your highest-resolution source. Use the table below:Source resolution
Pixels (W × H)
Minimum
SDK_VIDEO_SHM_BLOCK_SIZE(W × H × 3)1080p (default)
1920 × 1080
62208004K UHD
3840 × 2160
24883200DCI 4K
4096 × 2160
265420808K UHD
7680 × 4320
99532800Export the value (or set it in your
.env) before re-running your deployment mode:# Example: enable 4K ingestion (3840 x 2160 x 3 = 24883200 bytes per block) export SDK_VIDEO_SHM_BLOCK_SIZE=24883200 source setup.sh --search # re-run your mode so the new value applies
Budget the total shared memory. The pool pre-allocates
SDK_VIDEO_SHM_MAX_BLOCKS × SDK_VIDEO_SHM_BLOCK_SIZEbytes in the host/dev/shm. With the default512blocks this is ≈12.7 GBfor 4K and ≈51 GBfor 8K. If the host cannot spare that much (check withdf -h /dev/shm), lowerSDK_VIDEO_SHM_MAX_BLOCKSso the product fits:export SDK_VIDEO_SHM_BLOCK_SIZE=24883200 # 4K frame size export SDK_VIDEO_SHM_MAX_BLOCKS=128 # 128 x 24883200 ≈ 3.2 GB of /dev/shm source setup.sh --search
Note
Always size the block from the largest resolution you will ingest — an oversized block only wastes memory, while an undersized one triggers the failure above. For the full explanation, see the DataPrep Get Started guide.
Video Embedding Fails with “Pipeline made no progress”#
Problem: Embedding a video in --search or --dual mode fails after a few minutes, and the multimodal-dataprep logs show Pipeline made no progress for 300s while ... (MM_DATAPREP_PIPELINE_STALL_TIMEOUT_S=300). The video’s searchEmbeddings is false, unless an earlier run already succeeded.
Cause: A detection or embedding inference call in DataPrep never returned, usually because of a GPU hang. A stall watchdog fails the ingestion when no pipeline stage makes progress for MM_DATAPREP_PIPELINE_STALL_TIMEOUT_S seconds (default 300), instead of leaving it in progress forever.
Solution:
Check the host for GPU hang or reset messages with
sudo dmesg | grep -i -E "hang|reset", and restartmultimodal-dataprepto recover the device.If it recurs, run the affected stage on CPU: set
DATAPREP_DETECTION_DEVICE=CPUorDATAPREP_EMBEDDING_DEVICE=CPUand redeploy.Raise
MM_DATAPREP_PIPELINE_STALL_TIMEOUT_Sonly if slow hardware legitimately needs longer between stages;0disables the watchdog. On Helm, setmultimodaldataprep.sdkPipeline.pipelineStallTimeoutS.Retry with
POST /manager/videos/search-embeddings/{videoId}.
QwenText Embedding Crashes on Nova Lake CPUs#
Problem: On Intel® Core™ Ultra processors code-named Nova Lake, multimodal-embedding-serving (and multimodal-dataprep, which embeds in-process) crash as soon as a QwenText/qwen3-embedding-* model starts inference on CPU. The model downloads, converts and loads, then the worker dies; the service never becomes healthy. CLIP/* models are not affected.
Symptoms:
multimodal-embedding-servinglogs showQwen model loaded successfully, followed byWorker (pid:<N>) was sent SIGSEGV!andBooting worker with pid: <N>, repeating.--summary-and-search(unified) mode never finishes indexing summaries, and search returns no results.
Cause: A bug in the OpenVINO™ CPU plugin (releases up to 2026.4.1). Nova Lake processors support Intel® APX, and the CPU plugin’s dynamic-quantization kernel for INT8-compressed weights, which the QwenText models use, has a register conflict when APX is enabled. It is fixed upstream in openvinotoolkit/openvino#38042, planned for OpenVINO™ 2026.5. Other processors are not affected.
Solution: Run embedding on CPU with PyTorch instead of OpenVINO™. This is slower than OpenVINO™, so use it only on affected systems until a release built on OpenVINO™ 2026.5 is available.
Docker Compose: keep
DATAPREP_EMBEDDING_DEVICE,DATAPREP_DETECTION_DEVICEandMME_EMBEDDING_DEVICEat their defaultCPU(setup.shturns OpenVINO™ back on when any of them isGPUorNPU), then disable OpenVINO™ for embedding before starting the application:source setup.sh --down # if the application is already running export SDK_USE_OPENVINO=false export EMBEDDING_USE_OV=false # overrides a value left over from an earlier setup.sh run source setup.sh --summary-and-search
Helm: set both values in
user_values_override.yaml(themultimodalembeddingmsdefault ofembeddingUseOV: truetakes precedence over the global value), keep the embedding devices onCPU, and runhelm upgrade:global: env: SDK_USE_OPENVINO: false multimodalembeddingms: embeddingUseOV: false
To confirm the workaround, check that GET /model/current on multimodal-embedding-serving returns "use_openvino": false and that its logs no longer show SIGSEGV.
Accuracy of search results#
The accuracy of search results vary based on the embedding model used, configuration on frame sampling, object detection enabled or disabled, and the diversity of the video contents. The user is encouraged to check on these aspects in case the accuracy of the search results is not found to be satisfactory. Note that higher accuracy is normally a tradeoff with performance. Some specific pointers are provided below:
Model selection: Among the supported models, models with higher dimensionality will provide better results.
Higher frame sampling leads to better accuracy but at the cost of higher compute requirements.
Enabling object detection normally provides a better accuracy. Consider this option in alignment with selected model capability.
If the video diversity is very low, any query will seem to return the same results. Example: Same camera feed or video used for testing will return results from the same video irrespective of the query. Check the relevance score to determine how strong the match is.
Raise an issue in case of continued challenges faced.