# Troubleshooting

## Containers have started but the application is not working

You can try resetting the volume storage by deleting the previously created volumes:

**Note that this step does not apply when you are setting up the application for the first time.**

```bash
source setup.sh --clean-data
```

## OpenGL/Mesa Library Dependencies (Certain Kernel Versions)

On some Linux systems with certain kernel versions, you may encounter OpenCV-related errors due to missing OpenGL/Mesa libraries. If you experience issues with the summary stack or video processing, try installing the following dependencies:

```bash
sudo apt update
sudo apt install libgl1-mesa-dri libgl1-mesa-dev
```

After installing these dependencies:

1. Remove the `ov_models/` directory (if it exists)
2. Redeploy the VSS stack using the latest tagged images
3. Rerun your tests

This should resolve OpenCV-related dependency issues and allow the summary stack to work as expected.

## Search returns no results after changing embedding model

**Problem**: The UI displays `No videos found matching your search query. Try using different keywords or check if videos have been uploaded.` even though videos were ingested after running the setup script.

**Cause**: Either no videos have been processed yet, or the embedding model was switched to one with a different embedding dimension. Previously indexed vectors stay in the database, and their dimensions must match the active model. A mismatch prevents similarity lookups from returning any results.

**Solution**:

1. Verify at least one video has been uploaded or a summary run completed after the model change.
2. If you recently changed `MULTIMODAL_EMBEDDING_MODEL` or `TEXT_EMBEDDING_MODEL`, re-run ingestion so embeddings are recreated with the new dimensions. You can clean existing data with `source setup.sh --clean-data` and then bring the application back up with `source setup.sh --search`.
3. Review the supported embedding models and their dimensions in [Supported Models for Multimodal Embedding Serving](https://docs.openedgeplatform.intel.com/2026.2/edge-ai-libraries/multimodal-embedding-serving/supported-models.html) before switching models.

## VLM Microservice Model Loading Issues

**Problem**: VLM microservice fails to load or save models with permission errors, or you see errors related to model access in the logs.

**Cause**: This issue occurs when the `ov-models` Docker volume was created with incorrect ownership (root user) in previous versions of the application. The VLM microservice runs as a non-root user and requires proper permissions to read/write models.

**Symptoms**:

- VLM microservice container fails to start or crashes during model loading
- Permission denied errors in VLM service logs
- Model conversion or caching failures
- Error messages mentioning `/home/appuser/.cache/huggingface` or `/app/ov-model` access issues

**Solution**:

1. Stop the running application:

   ```bash
   source setup.sh --down
   ```

2. Remove the existing `ov-models` (old volume name) and `docker_ov-models` (updated volume name) Docker volume:

   ```bash
   docker volume rm ov-models docker_ov-models
   ```

3. Restart the application (the volume will be recreated with correct permissions):

   ```bash
   # For Video Summarization
   source setup.sh --summary

   # Or for Video Search
   source setup.sh --search
   ```

> **Note:** Removing the `ov-models` or `docker_ov-models` volume will delete any previously cached or converted models. The VLM service will automatically re-download and convert models on the next startup, which may take additional time depending on your internet connection and the model size.

**Prevention**: This issue has been fixed in the current version of the VLM microservice Dockerfile. New installations will automatically create the volume with correct permissions.

## VLM Final Summary Hallucination Issues

**Problem**: The final summary generated by the VLM microservice contains hallucinated or inaccurate information that does not reflect the actual video content.

**Cause**: This issue can occur when using smaller VLM models that may not have sufficient capacity to accurately process and summarize complex video content, leading to generation of plausible but incorrect information.

**Symptoms**:

- The final summary contains information not present in the video
- The Summary describes events, objects, or activities that do not actually occur in the video
- Inconsistent or contradictory information in the generated summary
- The Summary quality is poor despite chunk-wise summaries being accurate

**Solution**:
Try using a larger, more capable VLM model by updating the `VLM_MODEL_NAME` environment variable:

1. Stop the running application:

   ```bash
   source setup.sh --down
   ```

2. Set a larger VLM model (e.g., upgrade from 3B to 7B parameters):

   ```bash
   export VLM_MODEL_NAME="Qwen/Qwen2.5-VL-7B-Instruct"
   ```

3. Restart the application:

   ```bash
   source setup.sh --summary
   ```

**Alternative Models to Try**:

- For CPU: `Qwen/Qwen2.5-VL-7B-Instruct` (larger version)
- For GPU: Consider other supported VLM models with higher parameter counts

> **Note:** Larger models will require more system resources (RAM or VRAM) and may have longer inference times, but typically provide more accurate and coherent summaries.

## Final Summary Stuck or OVMS Container Stopped

**Problem**: The final video summary remains in a "Ready" or "In Progress" state indefinitely, and never completes.

**Cause**: The OVMS (OpenVINO™ Model Server) container may have crashed or the LLM request may have been rejected because the prompt size plus the requested `max_completion_tokens` exceeds the model's maximum context length. For example, if a model supports a 4096-token context window and the application requests 4000 completion tokens, even a modest prompt (~300 tokens) will exceed the limit.

**Symptoms**:

- Final summary status stays at "Ready" or "In Progress" and never progresses
- OVMS container has exited (shows as stopped in `docker ps -a`)
- OVMS logs contain errors like: `Number of prompt tokens: <N> + max tokens value: <M> exceeds model max length: <L>`
- OVMS logs contain `CL_OUT_OF_RESOURCES` or similar GPU memory errors

**Diagnosis**:

1. Check if the OVMS container is still running:

   ```bash
   docker ps -a | grep ovms
   ```

2. If the container has stopped or is in an exited state, check its logs:

   ```bash
   docker logs <ovms-container-name> 2>&1 | tail -50
   ```

3. Look for errors related to token limits or resource exhaustion in the log output.

**Solution**:

- If the logs show a **token limit exceeded** error, either reduce `SUMMARIZATION_MAX_COMPLETION_TOKENS` in your environment configuration, or switch to a model with a larger context window.
- If the logs show **GPU resource errors**, see the section below on GPU memory issues.
- After fixing the configuration, restart the application:

   ```bash
   source setup.sh --down
   source setup.sh --summary
   ```

## Smaller Models May Block Final Summary Due to Limited Context Window

**Problem**: The final video summary fails or hangs when using a smaller VLM/LLM model.

**Cause**: Smaller models often have a limited context window (e.g., 4096 tokens). When the combined prompt tokens and requested `max_completion_tokens` exceed this limit, the inference backend rejects the request and the final summary never completes.

**Symptoms**:

- Final summary status stays at "Ready" or "In Progress" indefinitely
- OVMS logs show errors such as: `Number of prompt tokens: <N> + max tokens value: <M> exceeds model max length: <L>`
- The chunk-wise summaries complete successfully but the final summary does not

**Solution**:

Reduce `PM_SUMMARIZATION_MAX_COMPLETION_TOKENS` to a value below the default of 4000 so that the prompt plus completion tokens fit within the model's context window:

```bash
export PM_SUMMARIZATION_MAX_COMPLETION_TOKENS=2000
source setup.sh --summary
```

Alternatively, switch to a model with a larger context window.

## VLM Workload Fails on NPU

**Problem**: The VLM model fails to load or run when `VLM_TARGET_DEVICE` or `LLM_TARGET_DEVICE` is set to `NPU`.

**Cause**: Not all VLM/LLM models are compatible with NPU execution. NPU support depends on the model architecture and the OpenVINO™ version installed.

**Symptoms**:

- OVMS container crashes or fails to start when targeting NPU
- Inference errors or unsupported-operation messages in OVMS logs
- Model conversion succeeds but inference produces errors

**Solution**:

1. Verify that your model is listed on the [OpenVINO™ Supported Models](https://docs.openvino.ai/2026/documentation/compatibility-and-support/supported-models.html) page for NPU execution.
2. If the model is not supported on NPU, switch to a supported model or fall back to CPU/GPU:

   ```bash
   export VLM_TARGET_DEVICE="CPU"
   source setup.sh --summary
   ```

## GPU Out-of-Resources When Loading Multiple Models

**Problem**: OVMS crashes or fails inference when multiple models (e.g., VLM + LLM) are loaded on the same GPU.

**Cause**: Loading multiple large models on a single GPU can exceed the available device memory. When the GPU runs out of resources during inference, the OpenCL runtime returns `CL_OUT_OF_RESOURCES` and OVMS terminates the request or crashes.

**Symptoms**:

- OVMS container exits unexpectedly or restarts repeatedly
- OVMS logs contain errors like:

  ```bash
  onednn_verbose,v1,primitive,error,ocl,errcode -5,CL_OUT_OF_RESOURCES
  Exception from src/plugins/intel_gpu/src/graph/impls/onednn/primitive_onednn_base.h
  Error occurred in LLM executor
  ```

- Inference requests hang and then fail
- Only one model works at a time but loading both causes failures

**Solution**:

1. **Distribute models across devices** — run the VLM on GPU and the LLM on CPU (or vice versa) to avoid competing for GPU memory. Adjust the device settings in your environment configuration accordingly.
2. **Use smaller model variants** — switch to quantized or smaller parameter models that consume less GPU memory.
3. **Increase GPU resources** — if available, use a GPU with more memory.
4. After making changes, restart the application:

   ```bash
   source setup.sh --down
   source setup.sh --summary
   ```

## Embedding Fails on NPU/GPU in Kubernetes (Device Permission)

**Problem**: On a Helm deployment, video embedding fails when `multimodal-embedding-ms` or `multimodal-dataprep` is configured for `NPU` (or `GPU`), even though the device plugin granted the resource and `/dev/accel` (or `/dev/dri`) is mounted in the pod. The UI may show a generic embedding failure.

**Cause**: The container runs as a non-root user. The accelerator device node (`/dev/accel` for NPU, `/dev/dri` for GPU) is owned by a host group (gid). Without that gid in the pod's `supplementalGroups`, the user cannot open the device, so OpenVINO™ enumerates only `['CPU']` and NPU/GPU plugin initialization fails — sometimes with a misleading "cannot load library …compiler_loader.so" message even though the library exists.

**Symptoms**:

- `multimodal-embedding-ms` / `multimodal-dataprep` logs show `DEVICES: ['CPU']` (NPU/GPU missing) and an NPU/GPU plugin/compiler load failure.
- The pod has the device mounted and the resource granted, yet inference stays on CPU or errors out.

**Solution**:

1. Find the gid that owns the device node on the scheduled worker node:

   ```bash
   ls -ln /dev/accel   # NPU
   ls -ln /dev/dri     # GPU
   ```

2. Set `global.accelGroupIds` in `user_values_override.yaml` to match (it is injected into the pod `supplementalGroups` when a service uses GPU/NPU):

   ```yaml
   global:
     accelGroupIds:
       - 992   # replace with the gid owning /dev/accel or /dev/dri on your node
   ```

3. Re-deploy (or `helm upgrade`) and confirm the pod now lists `['CPU','NPU']` (or `GPU`) and loads the model on the accelerator.

> **Note:** Nodes are heterogeneous, so the gid can differ per node. The chart cannot auto-detect it (templates render client-side), which is why `global.accelGroupIds` is an explicit override. On first GPU/NPU start the model is compiled and cached to `ovCacheDir` (`/app/ov_models/ov_cache`); allow the DataPrep `startupProbe` budget to complete this one-time compile.

## OVMS KV Cache Exhaustion

**Problem**: LLM or VLM inference requests are slow, produce incomplete responses, or fail under concurrent usage.

**Cause**: The OVMS KV cache is fully consumed. The KV cache holds intermediate attention state during text generation; when it is exhausted, OVMS must preempt or reject new requests.

**Symptoms**:

- OVMS logs show cache usage at or near 100 %:

  ```bash
  llm_executor.hpp:104] All requests: 1; Scheduled requests: 1; Cache type: static, cache usage: 100.0% of 4.0 GB;
  ```

- Requests take significantly longer than expected or time out
- Under concurrent requests, some are rejected or produce truncated output

**Diagnosis**:

1. Check OVMS container logs for cache usage lines:

   ```bash
   docker logs <ovms-container-name> 2>&1 | grep "cache usage"
   ```

2. If usage is consistently at or near **100 %**, the cache is too small for your workload.

**Solution**:

Increase the KV cache size by setting the `OVMS_CACHE_SIZE_GB` environment variable before running the setup script. The default is dynamically calculated based on available memory:

| Device | Allocation | Clamp range |
| ------ | ---------- | ----------- |
| CPU | 25 % of system RAM | [2, 16] GB |
| Integrated GPU (iGPU) | 25 % of system RAM | [2, 6] GB |
| Discrete GPU (dGPU) | 33 % of dedicated VRAM | [2, 16] GB |

The setup script automatically detects the GPU type using the OpenVINO™ runtime. Setting `OVMS_CACHE_SIZE_GB` overrides the dynamic calculation for all device types.

```bash
# Example: set KV cache to 8 GB
export OVMS_CACHE_SIZE_GB=8
source setup.sh --summary   # or --search
```

The updated cache size is applied to the existing model configuration on the next run — no re-export is required.

> **Note:** On integrated GPUs (iGPU), memory is shared with the system. Setting a very large cache size may leave insufficient memory for model weights and cause `CL_OUT_OF_RESOURCES` errors. Start with modest increases (e.g., 4 → 6 → 8 GB) and monitor both cache usage and GPU memory utilization.

## 4K/8K Video Ingestion Stalls with a Worker Timeout

**Problem**: Ingesting a high-resolution (4K/8K) video in `--search`, `--dual`, or `--unified` mode never completes. Search never returns results for that video, and the `multimodal-dataprep` container reboots its worker mid-ingestion.

**Cause**: The DataPrep decoder moves each frame between pipeline stages through a pool of fixed-size shared-memory blocks. Each decoded frame is written into a single block as a raw RGB buffer of exactly `width × height × 3` bytes. The block size is `SDK_VIDEO_SHM_BLOCK_SIZE`, which defaults to `6220800 = 1920 × 1080 × 3` (1080p). A 4K or 8K frame is several times larger than the default block, so the write fails, the frame is never enqueued, and every downstream stage (detection → embed → store → result) starves until the worker hits the Gunicorn timeout and is force-killed.

**Symptoms**:

- Every pipeline stage logs an empty queue, then the worker aborts with signal `134` (SIGABRT) and leaks shared-memory objects. Inspect with `docker logs multimodal-dataprep`:

  ```text
  WARNING: | detection_worker | [DETECTION QUEUE EMPTY] WAITING...
  WARNING: | store_worker      | [STORE_WORKER] Queue empty, waiting...
  WARNING: | embed_worker      | [EMBED_WORKER] Queue empty, waiting...
  WARNING: | process_result_worker | [RESULT WORKER] Queue empty, waiting...
  [CRITICAL] WORKER TIMEOUT (pid:8)
  [ERROR] Worker (pid:8) was sent code 134!
  UserWarning: resource_tracker: There appear to be 1024 leaked shared_memory objects to clean up at shutdown
  ```

- Only lower-resolution (1080p or smaller) videos index successfully.

**Solution**:

1. Set `SDK_VIDEO_SHM_BLOCK_SIZE` to at least `width × height × 3` for your highest-resolution source. Use the table below:

   | Source resolution | Pixels (W × H) | Minimum `SDK_VIDEO_SHM_BLOCK_SIZE` (`W × H × 3`) |
   | ----------------- | -------------- | ------------------------------------------------ |
   | 1080p (default)   | 1920 × 1080    | `6220800`                                        |
   | 4K UHD            | 3840 × 2160    | `24883200`                                       |
   | DCI 4K            | 4096 × 2160    | `26542080`                                       |
   | 8K UHD            | 7680 × 4320    | `99532800`                                       |

2. Export the value (or set it in your `.env`) **before** re-running your deployment mode:

   ```bash
   # Example: enable 4K ingestion (3840 x 2160 x 3 = 24883200 bytes per block)
   export SDK_VIDEO_SHM_BLOCK_SIZE=24883200
   source setup.sh --search   # re-run your mode so the new value applies
   ```

3. **Budget the total shared memory.** The pool pre-allocates `SDK_VIDEO_SHM_MAX_BLOCKS × SDK_VIDEO_SHM_BLOCK_SIZE` bytes in the host `/dev/shm`. With the default `512` blocks this is ≈ `12.7 GB` for 4K and ≈ `51 GB` for 8K. If the host cannot spare that much (check with `df -h /dev/shm`), lower `SDK_VIDEO_SHM_MAX_BLOCKS` so the product fits:

   ```bash
   export SDK_VIDEO_SHM_BLOCK_SIZE=24883200   # 4K frame size
   export SDK_VIDEO_SHM_MAX_BLOCKS=128        # 128 x 24883200 ≈ 3.2 GB of /dev/shm
   source setup.sh --search
   ```

> **Note:** Always size the block from the **largest** resolution you will ingest — an oversized block only wastes memory, while an undersized one triggers the failure above. For the full explanation, see the DataPrep [Get Started guide](https://github.com/open-edge-platform/edge-ai-libraries/blob/release-2026.2.0/microservices/visual-data-preparation-for-retrieval/multimodal-dataprep/docs/user-guide/get-started.md#advanced-tuning).

## Accuracy of search results

The accuracy of search results vary based on the embedding model used, configuration on frame sampling, object detection enabled or disabled, and the diversity of the video contents. The user is encouraged to check on these aspects in case the accuracy of the search results is not found to be satisfactory. Note that higher accuracy is normally a tradeoff with performance. Some specific pointers are provided below:

1. Model selection: Among the supported models, models with higher dimensionality will provide better results.
2. Higher frame sampling leads to better accuracy but at the cost of higher compute requirements.
3. Enabling object detection normally provides a better accuracy. Consider this option in alignment with selected model capability.
4. If the video diversity is very low, any query will seem to return the same results. Example: Same camera feed or video used for testing will return results from the same video irrespective of the query. Check the relevance score to determine how strong the match is.

Raise an issue in case of continued challenges faced.