# Release Notes: Video Search and Summarization Sample Application ## Version 2026.3.0-rc1 **Release Date:** November 6, 2026 > **Note:** Helm Chart support will be deprecated in the next release. Limited validation has been performed on the Helm charts for this release. **New:** - **MCP server for AI agents:** The VSS MCP tool server now starts with every Docker Compose deployment (`docker/compose.mcp.yaml`) and stops with `setup.sh --stop`; the Helm chart does not deploy it. The OpenAPI proxy was replaced with hand-declared tools that cover video listing, upload and indexing, text and image search, saved searches (create, poll, list, refetch and watch), summaries that follow the UI defaults unless the caller overrides them, live camera management, single-frame retrieval, tags and deployment info. - **Image search by reference:** Upload a query image once with `POST /search/images` (remove it with `DELETE /search/images/{imageId}`) and pass the returned `imageUrl` to `POST /search` instead of inline base64. Hits returned by `GET /search/{queryId}` include a playable link to the matching video; set `PM_PUBLIC_BASE_URL` (Helm `pipelinemanager.env.PM_PUBLIC_BASE_URL`) when the gateway is reached through another host name. - **Tags on live cameras:** Tags set on a camera when it is added or updated are registered with Pipeline Manager, so they appear in `GET /tags` and in the search tag filters right away. - **Refresh cached images on deploy:** `setup.sh --pull-always` (or `PULL_ALWAYS=true`, `make deploy PULL_ALWAYS=true`) pulls newer images for moving tags such as `latest` instead of reusing a cached copy. - **Live RTSP stream ingestion:** Register RTSP cameras from the new **Live Streams** panel in the search UI and ingest them continuously into the search index. Cameras can be listed, filtered by state or tag, paused, resumed, reconfigured, and removed, with live frame and embedding counts refreshed over the existing websocket. Backed by new Pipeline Manager `/streams` endpoints that proxy the Multimodal DataPrep live-stream API. Available in `--search` and `--dual` (`--summary --search`) modes, but not in `--summary-and-search`/`--unified`, where the index holds text embeddings of summaries; enable or disable the UI with `UI_LIVE_STREAMS_FEATURE` and bound index growth with `LIVE_RETENTION_HOURS` (defaults to 24 hours). RTSP credentials are redacted from all API responses, logs, and search results. - **Minimum peak score filter:** Search can drop weak matches at retrieval: frames scoring below `minPeakScore` (0-1) are filtered in the Vector Retriever before aggregation, so every returned result's best frame scores at least that value. Set it per search in the UI (off by default; starts at 0.2 when turned on), on `POST /search`, `POST /search/query`, `POST /search/{queryId}/refetch` and `PATCH /search/{queryId}/watch`, or as the deployment default with `SEARCH_MIN_PEAK_SCORE` (default `0`, off). The value is saved with each query, so watched-query refreshes are gated by it. The MCP `vss_search_video` and `vss_get_search` tools accept it as `min_peak_score`, and a watched search keeps the value saved on it. - **Sort and group search results by time:** Pipeline Manager now dates every search result with `observedAt` and `timeSource` (`live`: camera capture time; `upload`: the video's upload time plus the frame offset) and adds `streamId` to live hits. The search UI adds **Sort by** (Relevance, Newest first, Oldest first) and **Group by** (None, Tag, Camera / video) controls, replacing the **Group by Tag** button, and shows each result's capture time, or its upload time with the match's offset into the video (hits from one video sort by that offset). The MCP `vss_search_video` and `vss_get_search` tools accept `sort` (`relevance`, `newest`, `oldest`) and return `observed_at`/`time_source` on every hit. Sorting reorders the top results by relevance; it does not fetch lower-ranked moments. - **Geo Spatial Map View:** **Map View** in the search results header shows results as pins on a map beside the results, one per location, with its clip count. Off by default; enable it with `APP_MAP_FEATURE=FEATURE_ON` and see [Geo Spatial Map View](./geo-spatial-map-view.md). - **Re-add recent uploads:** Several videos can be picked from the recent uploads list in the search UI; each is added again as a new video with the same name and the copies are embedded in one batch job. Backed by the new `POST /videos/{videoId}/copy` endpoint, which copies the stored file to a new `videoId` inside the object store. **Improved:** - **Dependency refresh:** Updated the Pipeline Manager and UI dependency sets to the latest in-range patch/minor releases. UI advisories drop from 6 to **0**; Pipeline Manager drops from 19 to **8**. The residual Pipeline Manager advisories come from three transitive packages, each unfixable without causing a regression: - `braces` (via the `@swc/cli` build tool, dev-only) has no patched release — every published version is affected. - `stream-json` and `decode-uri-component` arrive through `minio@8.0.7`, the latest release. `npm audit fix --force` "fixes" them by **downgrading** minio to `7.1.3`, and minio only imports `stream-json/jsonl/Parser.js`, not the vulnerable `pick`/`ignore`/`filter`/`replace` filters named in the advisory. Overriding `decode-uri-component` to the patched `0.5.0` is also not viable: that release is ESM-only and would make minio's CommonJS build throw `ERR_REQUIRE_ESM` at runtime. - **Dependency override cleanup:** Removed 9 now-redundant `overrides` entries from Pipeline Manager (`sha.js`, `qs`, `systeminformation`, `fast-xml-parser`, `socket.io-parser`, `lodash`, `body-parser`, `multer`, `serialize-javascript`) and all 9 from the UI, after confirming each one no longer changes dependency resolution. The remaining Pipeline Manager overrides are retained because dropping them measurably readmits older duplicate copies (`ajv`, `minimatch`, `picomatch`, `glob`, `webpack`) or downgrades a package outright (`diff` 8.0.4 → 4.0.4). - **Live camera limit is configurable:** `LIVE_STREAM_MAX_CONCURRENT` (default `8`; Helm `multimodaldataprep.liveStream.maxConcurrent`) sets how many cameras ingest at once. Registering more returns `503`; pause or delete a camera to free a slot. - **Search filters panel:** The time range, number of results and minimum peak score filters share one panel. Edits are drafts until **Apply**, and sliders show their value while dragging. - **Result tags in every view:** Each search result shows its tags with **Group by** set to **None** as well, and each tag gets a stable colour derived from its name, dark enough for white text. - **OVMS 2026.4.1:** Docker Compose and Helm now use OpenVINO™ Model Server `2026.4.1`, and Pipeline Manager waits for a healthy OVMS whenever OVMS serves the VLM or LLM, so a deployment fails visibly instead of running without a model server. OVMS `max_num_seqs` is sized from the Pipeline Manager concurrency (at least 16; override with `OVMS_MAX_NUM_SEQS`), and `OVMS_CACHE_SIZE_GB` is passed through. Pre-converted `OpenVINO/*` models take their weight format from the repository name and are checked against the target device before download. - **NPU LLM prompt limits:** An LLM on NPU is compiled with `OVMS_NPU_MAX_PROMPT_LEN` (default `8192`) and `OVMS_NPU_MIN_RESPONSE_LEN` (default `1024`), and Pipeline Manager's context and completion budgets are derived from them, so long summary prompts are no longer rejected by the NPU. A vision-language model set as the NPU LLM is rejected at setup. - **Inference concurrency defaults:** With OVMS, `setup.sh` defaults to `PM_VLM_CONCURRENT=4` and `PM_LLM_CONCURRENT=2` on every device (Helm `pipelinemanager.env.VLM_CONCURRENT` `6`, `LLM_CONCURRENT` `2`); vLLM still uses `1`. - **Faster startup detection:** The Pipeline Manager container healthcheck polls every 5 seconds with a longer retry budget, so dependent services start sooner and a slow first start is not reported as unhealthy. - **Platform updates:** Video Ingestion uses the `intel/dlstreamer-pipeline-server:2026.3.0-20261027-weekly-ubuntu24` base image, the summarizer CLI requires Go 1.26, and the Multimodal Embedding Serving and DataPrep images install the Intel® GPU compute runtime directly in the Dockerfile and fail the build if a driver is missing. Python, Node.js and Go dependencies were refreshed to clear fixable image vulnerabilities in VSS-owned code; remaining findings come from base images. - The browser tab title no longer carries the "Intel EGAI" prefix. - **DataPrep object detection uses Hugging Face OpenVINO models:** Set `DATAPREP_DETECTION_MODEL` (Helm `multimodaldataprep.frameProcessing.detectionModelName`) to an `OpenVINO/` object-detection model, for example `OpenVINO/yolox_s-int8-ov`; `OpenVINO/yolox_tiny-fp16-ov`, `OpenVINO/yolox_l-int8-ov` and `OpenVINO/YOLO11n-fp16-ov` are also verified on CPU, GPU and NPU. It has no default and is required in `--search` and `--summary --search` mode unless `ENABLE_OBJECT_DETECTION=false`; `--summary-and-search` mode always runs without detection. The optional `DATAPREP_DETECTION_CLASSES` (Helm `detectionClasses`) keeps only the listed classes. Models are cached in the `docker_dataprep-detection-models` volume; the old `docker_dataprep-yolox-models` volume is no longer used and can be removed. **Fixed:** - **Video embedding status is recorded:** `GET /videos` and `GET /videos/{videoId}` always returned `searchEmbeddings` and `textEmbeddings` as `null`, so clients, including the MCP server, could not tell whether a video was indexed. Pipeline Manager now records them from single and batch frame-embedding requests and from summary caption embedding; a `true` flag is never downgraded. The unused `videoEmbeddings` field was removed. Live cameras have no video record and keep reporting progress through `GET /streams`. - **Stalled ingestion no longer hangs:** A DataPrep video ingestion whose detection or embedding inference never returns (for example after a GPU hang) now fails after `MM_DATAPREP_PIPELINE_STALL_TIMEOUT_S` seconds without progress (default `300`, `0` disables it; Helm `multimodaldataprep.sdkPipeline.pipelineStallTimeoutS`), instead of staying in progress forever. - **Blank captions and summaries with reasoning models on OVMS:** Reasoning-capable models could spend the whole completion budget thinking and return empty content. Pipeline Manager now disables thinking (`chat_template_kwargs.enable_thinking=false`) on OVMS requests; models that do not use the flag are unaffected. - **`make push` skipped the MCP image:** `make push` now pushes `vss-mcp-server` with the same name `make build` uses. **Upgrade Notes:** - **Breaking:** In `--search` and `--summary --search` mode, `DATAPREP_DETECTION_MODEL` must be set (or `ENABLE_OBJECT_DETECTION=false`); GitHub YOLOX names such as `yolox_s` are no longer accepted. On Helm, set `multimodaldataprep.frameProcessing.detectionModelName`. - Each detection model keeps its own license (YOLOX is Apache-2.0, YOLO11 is AGPL-3.0); review the model card before use. - Building the summarizer CLI needs Go 1.26 or later. **Known Issues:** - **QwenText embedding crashes on Intel® Core™ Ultra processors (Nova Lake) on CPU:** `multimodal-embedding-serving` and `multimodal-dataprep` crash with `SIGSEGV` on the first inference when a `QwenText/qwen3-embedding-*` model (the default for `--summary-and-search` mode) runs on CPU with OpenVINO™. Gunicorn logs `Worker (pid:) was sent SIGSEGV!` and restarts the worker in a loop. This is an OpenVINO™ CPU plugin bug in releases up to 2026.4.1, fixed upstream for 2026.5. As a workaround, run embedding on CPU without OpenVINO™; see [QwenText Embedding Crashes on Nova Lake CPUs](./troubleshooting.md#qwentext-embedding-crashes-on-nova-lake-cpus). --- ## Version 2026.2.0 **Release Date:** September 9, 2026 **New:** - **Backend-agnostic vector search:** Select the vector database with `VECTORDB_BACKEND` (`vdms` default, `milvus`). All vector similarity search is delegated to the standalone `vector-retriever` microservice; `video-search` keeps query orchestration and frame-to-video aggregation, so results are identical in shape across backends. - **Multimodal DataPrep:** Replaced the legacy VDMS DataPrep with the `multimodal-dataprep` microservice across Docker Compose, Helm, and documentation. - **Metrics Manager integration:** Replaced the legacy telemetry collector; the search UI now shows live multi-GPU telemetry (`ENABLE_METRICS_MANAGER=true`). - **Batch video upload and embedding:** Multi-select upload in the UI uploads all files and submits a single asynchronous batch embeddings job, with new `search-embeddings-batch` and `search-embeddings-jobs` Pipeline Manager endpoints and job-status polling. Single-file upload keeps the synchronous path. - **Directory watcher batch processing:** The directory watcher service now batches video uploads and embedding creation. - **Duplicate-content upload policy:** New `MM_DATAPREP_ALLOW_DUPLICATE_UPLOADS` setting (Docker Compose and Helm `global.env`) rejects content-identical re-uploads with HTTP `409`. Enforced per item for batch-processed media, so a duplicate fails only its own item. - **Model download support:** Models are now fetched by a dedicated model-download step in `setup.sh`, including generic YOLO ids from the ultralytics hub for object detection. - **Search score transparency:** Search results show the query-normalized relevance score alongside the underlying raw segment score and peak frame similarity, with a per-stage scoring breakdown popover and a weak-match indicator. - **Makefile, functional test suite, and CI security scans** for the Video Search and Summarization sample application. - **AI agent skills:** Added VSS agent skills (deploy, build, troubleshoot, summarize, search, and more) for AI coding assistants. **Improved:** - **Search orchestration alignment for accelerator usage:** Updated setup and compose behavior for Video Search so NPU device selections are preserved and propagated consistently for DataPrep and multimodal embedding services. - **Simplified per-component device model (Compose):** Retired the redundant `VDMS_DATAPREP_DEVICE` baseline knob. Device selection is now purely per-component — `DATAPREP_EMBEDDING_DEVICE`, `DATAPREP_DETECTION_DEVICE`, and `MME_EMBEDDING_DEVICE` (each defaults to `CPU`) — matching the Helm chart model. `ENABLE_EMBEDDING_GPU` is now a mode-aware embedding shortcut (`sdk`→DataPrep embedding GPU, `api`→MME embedding GPU). - **Helm accelerator support for search stack:** Updated VSS Helm subcharts for `multimodal-embedding-ms` and `multimodal-dataprep` to support NPU as an accelerator path (device-key validation, resource requests/limits, and `/dev/accel` mounts). - **Helm accelerator device permissions:** Added `global.accelGroupIds` so the host gids owning `/dev/dri` (GPU) and `/dev/accel` (NPU) are injected into the pod `supplementalGroups`, letting the non-root container open the accelerator device (mirrors the Compose `group_add` render/video groups). Fixes NPU/GPU device initialization falling back to CPU-only. - **Helm OpenVINO™ model cache:** Added a persistent OpenVINO™ cache (`ovCacheDir`, default `/app/ov_models/ov_cache`) for `multimodal-embedding-ms` and `multimodal-dataprep`, plus a longer DataPrep `startupProbe` budget, so GPU/NPU model compilation completes once and is reused across pod restarts instead of recompiling (avoids startup crash loops). - **Helm single-source image override:** `global.registry`, `global.tag`, and `global.pullPolicy` now apply across all VSS service images (pipeline-manager, video-ingestion, video-search, vss-ui, multimodal-dataprep, multimodal-embedding-serving) from one place, with independent per-service PVCs for model/cache data. - **Helm value override precedence:** Subchart defaults no longer shadow `global.env` overrides for `MM_DATAPREP_ALLOW_DUPLICATE_UPLOADS` and `SDK_USE_OPENVINO`, so `--set global.env.` now takes effect. - **Clearer embedding error reporting:** The video embedding flow now surfaces the real upstream DataPrep error instead of a misleading "Request timed out" message; only genuine timeouts (`408`/`504`/connection aborts) are reported as timeouts. - **Graceful OVMS cache fallback:** `setup.sh` now degrades cleanly when GPU drivers are absent instead of failing model cache setup. - **Original filenames in embedding metadata:** The uploaded video name is threaded end-to-end into stored embedding metadata, so search results show the original filename. - **Service health and startup dependencies** hardened across the stack, with retries for transient batch job polling failures. - **Search deployment documentation refresh:** Added a dedicated **Deployment Options for Video Search** matrix (SDK/API with CPU/GPU/NPU combinations), including explicit `DATAPREP_EMBEDDING_DEVICE`, `MME_EMBEDDING_DEVICE`, and `DATAPREP_DETECTION_DEVICE` examples for accelerator-specific routing. - **Helm user-guide clarifications:** Updated Helm guidance to include NPU device/key combinations and matching-device recommendations for shared PVC scheduling. - **Documentation:** Aligned user and agent documentation with the multi-backend architecture, refreshed the DataPrep, retriever, and VSS OpenAPI specs, added 4K/8K video ingestion troubleshooting guidance, and split 2025 release notes into a separate document. - **Maintenance:** Refreshed dependency version constraints and lock files, pinned Dockerfiles to release versions, aligned the DataPrep bucket with the video summary bucket, updated the VSS health check with a manual CI workflow trigger, and removed the `profile_dataprep` script and the vestigial `EMBEDDING_PROCESSING_MODE` chart value. **Fixed:** - Duplicate-upload conflicts no longer leave orphaned video tiles in the UI. - Search result tiles no longer render as black thumbnails. - Assorted `search-ms` and video embedding flow defects. ## Version 2026.1.0 **Release Date:** June 17, 2026 **New:** - **EXPERIMENTAL - vLLM Intel® Arc™ Pro B-series GPU Support:** Added initial experimental support for running vLLM on Intel® Arc™ Pro B-series GPUs (XPUs) via new `ENABLE_VLLM_GPU` environment variable and `docker/compose.vllm.xpu.yaml` Docker Compose overlay. This feature enables GPU-accelerated VLM captioning and LLM summarization on Intel® Arc™ Pro B-series hardware (e.g., B60, B65, B70). > [!NOTE] > This is an experimental feature in early stages and may require additional tuning and optimization. Performance characteristics are still being evaluated. Not recommended for production use. - Added a a new Dual UI mode with a new `--summary --search` CLI argument for `setup.sh` that allows running both the summary and the search applications simultaneously at **/summary** and **/search** URI endpoints respectively. - Added Dual UI support for Helm chart installations by allowing a values override file to be provided for summary and search modes simultaneously. **Improved:** - Updated setup script and nginx configuration files to allow flexible UI routing for each existing mode of deployment (summary mode, search mode, Unified UI Mode) and the new Dual UI mode. - Refactored Helm chart to use a reusable `vssui` subchart with multi-mode nginx and consolidated embedding model config under `global.embeddingModelName`. - Updated DL Streamer base image to `2026.1.0-ubuntu24-rc1` for Video Ingestion Microservice. - **Setup Script:** Updated the environment variable to setup embedding models. New `MULTIMODAL_EMBEDDING_MODEL` and the existing `TEXT_EMBEDDING_MODEL` are used to provide embedding models in relevant modes. - **Docker Compose:** Replaced `curl` with Python `urllib` package in the container `healthcheck` command for a lighter runtime footprint for Audio Analyzer. - **Docker Compose:** Replaced environment variables with hard-coded mount paths. This helps in stopping containers without looking for preset variables. - **Build Script:** Removed Audio-Analyzer from the dependency build pipeline. A frozen version 1.3.3 will be used for the Audio Analyzer microservice for the current and all subsequent releases. - **Setup Script:** Removed unused environment variables and several environment variables being used as mount directories in Docker Compose files. ## Version 1.3.3-rc1 **Release Date:** 05 May 2026 **Features:** - **Configurable final video summary**: Added `PM_PRODUCE_FINAL_SUMMARY` feature flag to make the final LLM map-reduce video summary optional. When disabled, chunk-wise summaries are displayed chronologically instead. A per-video UI override checkbox is available in both upload flows. Audio transcript summarization is automatically skipped when the final summary is turned off. - **Audio transcript summarization**: Added audio transcript summarization support and improved audio transcription accuracy. - **OVMS-first architecture**: Replaced the standalone `vlm-openvino-serving` microservice with OpenVINO™ Model Server (OVMS) as the unified inference backend for both VLM captioning and LLM summarization. This is a **breaking change**; the `vlm-inference` subchart and container have been removed. - **Performance Optimizations (MME & VDMS-Data-Prep)**: - Refactored pre-processing and inference with `AsyncInferQueue` based OpenVINO™ inference and static shape model compilation for iGPU. - Added ThreadPool for parallel open_clip image pre-processing with support for input tensor batching and padding for optimal OpenVINO™ inference paths. - Introduced PyAV-based video decode abstraction supporting keyframes and uniform sampled frames extraction with producer-consumer pattern for parallel decode and frame translation to PIL. - Enabled multiple/parallel decoder instances for file, RTSP stream, and bytes input sources. - Implemented frame batching for pipelined pre-processing and inference with integrated PyAV decoder in VDMS data-prep. - **Search Timeout and Resource Management**: Added `SEARCH_DATAPREP_TIMEOUT_MS` configuration to prevent VSS-UI timing out during embedding creation. Added ulimit constraints with soft and hard limits to enable shared memory creation and define memory block allocation boundaries. **HW used for validation:** - Intel® Xeon® 5 + Intel® Arc™ B580 GPU - Vanilla Kubernetes Cluster **Known Issues/Limitations:** - This release includes only limited testing on EMT‑S and EMT‑D, some behaviors may not yet be fully validated across all scenarios. - HW sizing of the Video Search or Video Summarization pipeline is in progress. Optimization of the pipelines will follow HW sizing. - Known issues are internally tracked. Reference not provided here. - `how-to-performance` document is not updated yet. HW sizing details will be added to this section shortly. - NPU support with OVMS is added as experimental feature and may not work for all models or configurations. ## Version 1.3.2 **Release Date:** 17 Feb 2026 **Features:** - In VSS search mode, users can now filter results by time range via: - Query parsing to infer time ranges (e.g., "person seen in last 5 minutes"). - Direct time range input from the UI. - Added live system and DataPrep performance metrics in the search UI (enable with `export ENABLE_METRICS_MANAGER=true`). - Fixed the build script of the `vdms-dataprep` microservice. - Added telemetry collection of the application metrics for VDMS-dataprep microservice and VLM microservice at `/telemetry` endpoint. **HW used for validation:** - Intel® Xeon® 5 + Intel® Arc™ B580 GPU - Vanilla Kubernetes Cluster ## Previous Releases - [Release Notes 2025](./release-notes/release-notes-2025.md) :::{toctree} :hidden: Release Notes 2025 :::