Release Notes: Video Search and Summarization Sample Application#

Version 2026.3.0-rc1#

Release Date: November 6, 2026

Note: Helm Chart support will be deprecated in the next release. Limited validation has been performed on the Helm charts for this release.

New:

  • MCP server for AI agents: The VSS MCP tool server now starts with every Docker Compose deployment (docker/compose.mcp.yaml) and stops with setup.sh --stop; the Helm chart does not deploy it. The OpenAPI proxy was replaced with hand-declared tools that cover video listing, upload and indexing, text and image search, saved searches (create, poll, list, refetch and watch), summaries that follow the UI defaults unless the caller overrides them, live camera management, single-frame retrieval, tags and deployment info.

  • Image search by reference: Upload a query image once with POST /search/images (remove it with DELETE /search/images/{imageId}) and pass the returned imageUrl to POST /search instead of inline base64. Hits returned by GET /search/{queryId} include a playable link to the matching video; set PM_PUBLIC_BASE_URL (Helm pipelinemanager.env.PM_PUBLIC_BASE_URL) when the gateway is reached through another host name.

  • Tags on live cameras: Tags set on a camera when it is added or updated are registered with Pipeline Manager, so they appear in GET /tags and in the search tag filters right away.

  • Refresh cached images on deploy: setup.sh --pull-always (or PULL_ALWAYS=true, make deploy PULL_ALWAYS=true) pulls newer images for moving tags such as latest instead of reusing a cached copy.

  • Live RTSP stream ingestion: Register RTSP cameras from the new Live Streams panel in the search UI and ingest them continuously into the search index. Cameras can be listed, filtered by state or tag, paused, resumed, reconfigured, and removed, with live frame and embedding counts refreshed over the existing websocket. Backed by new Pipeline Manager /streams endpoints that proxy the Multimodal DataPrep live-stream API. Available in --search and --dual (--summary --search) modes, but not in --summary-and-search/--unified, where the index holds text embeddings of summaries; enable or disable the UI with UI_LIVE_STREAMS_FEATURE and bound index growth with LIVE_RETENTION_HOURS (defaults to 24 hours). RTSP credentials are redacted from all API responses, logs, and search results.

  • Minimum peak score filter: Search can drop weak matches at retrieval: frames scoring below minPeakScore (0-1) are filtered in the Vector Retriever before aggregation, so every returned result’s best frame scores at least that value. Set it per search in the UI (off by default; starts at 0.2 when turned on), on POST /search, POST /search/query, POST /search/{queryId}/refetch and PATCH /search/{queryId}/watch, or as the deployment default with SEARCH_MIN_PEAK_SCORE (default 0, off). The value is saved with each query, so watched-query refreshes are gated by it. The MCP vss_search_video and vss_get_search tools accept it as min_peak_score, and a watched search keeps the value saved on it.

  • Sort and group search results by time: Pipeline Manager now dates every search result with observedAt and timeSource (live: camera capture time; upload: the video’s upload time plus the frame offset) and adds streamId to live hits. The search UI adds Sort by (Relevance, Newest first, Oldest first) and Group by (None, Tag, Camera / video) controls, replacing the Group by Tag button, and shows each result’s capture time, or its upload time with the match’s offset into the video (hits from one video sort by that offset). The MCP vss_search_video and vss_get_search tools accept sort (relevance, newest, oldest) and return observed_at/time_source on every hit. Sorting reorders the top results by relevance; it does not fetch lower-ranked moments.

  • Geo Spatial Map View: Map View in the search results header shows results as pins on a map beside the results, one per location, with its clip count. Off by default; enable it with APP_MAP_FEATURE=FEATURE_ON and see Geo Spatial Map View.

  • Re-add recent uploads: Several videos can be picked from the recent uploads list in the search UI; each is added again as a new video with the same name and the copies are embedded in one batch job. Backed by the new POST /videos/{videoId}/copy endpoint, which copies the stored file to a new videoId inside the object store.

Improved:

  • Dependency refresh: Updated the Pipeline Manager and UI dependency sets to the latest in-range patch/minor releases. UI advisories drop from 6 to 0; Pipeline Manager drops from 19 to 8. The residual Pipeline Manager advisories come from three transitive packages, each unfixable without causing a regression:

    • braces (via the @swc/cli build tool, dev-only) has no patched release — every published version is affected.

    • stream-json and decode-uri-component arrive through minio@8.0.7, the latest release. npm audit fix --force “fixes” them by downgrading minio to 7.1.3, and minio only imports stream-json/jsonl/Parser.js, not the vulnerable pick/ignore/filter/replace filters named in the advisory. Overriding decode-uri-component to the patched 0.5.0 is also not viable: that release is ESM-only and would make minio’s CommonJS build throw ERR_REQUIRE_ESM at runtime.

  • Dependency override cleanup: Removed 9 now-redundant overrides entries from Pipeline Manager (sha.js, qs, systeminformation, fast-xml-parser, socket.io-parser, lodash, body-parser, multer, serialize-javascript) and all 9 from the UI, after confirming each one no longer changes dependency resolution. The remaining Pipeline Manager overrides are retained because dropping them measurably readmits older duplicate copies (ajv, minimatch, picomatch, glob, webpack) or downgrades a package outright (diff 8.0.4 → 4.0.4).

  • Live camera limit is configurable: LIVE_STREAM_MAX_CONCURRENT (default 8; Helm multimodaldataprep.liveStream.maxConcurrent) sets how many cameras ingest at once. Registering more returns 503; pause or delete a camera to free a slot.

  • Search filters panel: The time range, number of results and minimum peak score filters share one panel. Edits are drafts until Apply, and sliders show their value while dragging.

  • Result tags in every view: Each search result shows its tags with Group by set to None as well, and each tag gets a stable colour derived from its name, dark enough for white text.

  • OVMS 2026.4.1: Docker Compose and Helm now use OpenVINO™ Model Server 2026.4.1, and Pipeline Manager waits for a healthy OVMS whenever OVMS serves the VLM or LLM, so a deployment fails visibly instead of running without a model server. OVMS max_num_seqs is sized from the Pipeline Manager concurrency (at least 16; override with OVMS_MAX_NUM_SEQS), and OVMS_CACHE_SIZE_GB is passed through. Pre-converted OpenVINO/* models take their weight format from the repository name and are checked against the target device before download.

  • NPU LLM prompt limits: An LLM on NPU is compiled with OVMS_NPU_MAX_PROMPT_LEN (default 8192) and OVMS_NPU_MIN_RESPONSE_LEN (default 1024), and Pipeline Manager’s context and completion budgets are derived from them, so long summary prompts are no longer rejected by the NPU. A vision-language model set as the NPU LLM is rejected at setup.

  • Inference concurrency defaults: With OVMS, setup.sh defaults to PM_VLM_CONCURRENT=4 and PM_LLM_CONCURRENT=2 on every device (Helm pipelinemanager.env.VLM_CONCURRENT 6, LLM_CONCURRENT 2); vLLM still uses 1.

  • Faster startup detection: The Pipeline Manager container healthcheck polls every 5 seconds with a longer retry budget, so dependent services start sooner and a slow first start is not reported as unhealthy.

  • Platform updates: Video Ingestion uses the intel/dlstreamer-pipeline-server:2026.3.0-20261027-weekly-ubuntu24 base image, the summarizer CLI requires Go 1.26, and the Multimodal Embedding Serving and DataPrep images install the Intel® GPU compute runtime directly in the Dockerfile and fail the build if a driver is missing. Python, Node.js and Go dependencies were refreshed to clear fixable image vulnerabilities in VSS-owned code; remaining findings come from base images.

  • The browser tab title no longer carries the “Intel EGAI” prefix.

  • DataPrep object detection uses Hugging Face OpenVINO models: Set DATAPREP_DETECTION_MODEL (Helm multimodaldataprep.frameProcessing.detectionModelName) to an OpenVINO/<repo> object-detection model, for example OpenVINO/yolox_s-int8-ov; OpenVINO/yolox_tiny-fp16-ov, OpenVINO/yolox_l-int8-ov and OpenVINO/YOLO11n-fp16-ov are also verified on CPU, GPU and NPU. It has no default and is required in --search and --summary --search mode unless ENABLE_OBJECT_DETECTION=false; --summary-and-search mode always runs without detection. The optional DATAPREP_DETECTION_CLASSES (Helm detectionClasses) keeps only the listed classes. Models are cached in the docker_dataprep-detection-models volume; the old docker_dataprep-yolox-models volume is no longer used and can be removed.

Fixed:

  • Video embedding status is recorded: GET /videos and GET /videos/{videoId} always returned searchEmbeddings and textEmbeddings as null, so clients, including the MCP server, could not tell whether a video was indexed. Pipeline Manager now records them from single and batch frame-embedding requests and from summary caption embedding; a true flag is never downgraded. The unused videoEmbeddings field was removed. Live cameras have no video record and keep reporting progress through GET /streams.

  • Stalled ingestion no longer hangs: A DataPrep video ingestion whose detection or embedding inference never returns (for example after a GPU hang) now fails after MM_DATAPREP_PIPELINE_STALL_TIMEOUT_S seconds without progress (default 300, 0 disables it; Helm multimodaldataprep.sdkPipeline.pipelineStallTimeoutS), instead of staying in progress forever.

  • Blank captions and summaries with reasoning models on OVMS: Reasoning-capable models could spend the whole completion budget thinking and return empty content. Pipeline Manager now disables thinking (chat_template_kwargs.enable_thinking=false) on OVMS requests; models that do not use the flag are unaffected.

  • make push skipped the MCP image: make push now pushes vss-mcp-server with the same name make build uses.

Upgrade Notes:

  • Breaking: In --search and --summary --search mode, DATAPREP_DETECTION_MODEL must be set (or ENABLE_OBJECT_DETECTION=false); GitHub YOLOX names such as yolox_s are no longer accepted. On Helm, set multimodaldataprep.frameProcessing.detectionModelName.

  • Each detection model keeps its own license (YOLOX is Apache-2.0, YOLO11 is AGPL-3.0); review the model card before use.

  • Building the summarizer CLI needs Go 1.26 or later.

Known Issues:

  • QwenText embedding crashes on Intel® Core™ Ultra processors (Nova Lake) on CPU: multimodal-embedding-serving and multimodal-dataprep crash with SIGSEGV on the first inference when a QwenText/qwen3-embedding-* model (the default for --summary-and-search mode) runs on CPU with OpenVINO™. Gunicorn logs Worker (pid:<N>) was sent SIGSEGV! and restarts the worker in a loop. This is an OpenVINO™ CPU plugin bug in releases up to 2026.4.1, fixed upstream for 2026.5. As a workaround, run embedding on CPU without OpenVINO™; see QwenText Embedding Crashes on Nova Lake CPUs.


Version 2026.2.0#

Release Date: September 9, 2026

New:

  • Backend-agnostic vector search: Select the vector database with VECTORDB_BACKEND (vdms default, milvus). All vector similarity search is delegated to the standalone vector-retriever microservice; video-search keeps query orchestration and frame-to-video aggregation, so results are identical in shape across backends.

  • Multimodal DataPrep: Replaced the legacy VDMS DataPrep with the multimodal-dataprep microservice across Docker Compose, Helm, and documentation.

  • Metrics Manager integration: Replaced the legacy telemetry collector; the search UI now shows live multi-GPU telemetry (ENABLE_METRICS_MANAGER=true).

  • Batch video upload and embedding: Multi-select upload in the UI uploads all files and submits a single asynchronous batch embeddings job, with new search-embeddings-batch and search-embeddings-jobs Pipeline Manager endpoints and job-status polling. Single-file upload keeps the synchronous path.

  • Directory watcher batch processing: The directory watcher service now batches video uploads and embedding creation.

  • Duplicate-content upload policy: New MM_DATAPREP_ALLOW_DUPLICATE_UPLOADS setting (Docker Compose and Helm global.env) rejects content-identical re-uploads with HTTP 409. Enforced per item for batch-processed media, so a duplicate fails only its own item.

  • Model download support: Models are now fetched by a dedicated model-download step in setup.sh, including generic YOLO ids from the ultralytics hub for object detection.

  • Search score transparency: Search results show the query-normalized relevance score alongside the underlying raw segment score and peak frame similarity, with a per-stage scoring breakdown popover and a weak-match indicator.

  • Makefile, functional test suite, and CI security scans for the Video Search and Summarization sample application.

  • AI agent skills: Added VSS agent skills (deploy, build, troubleshoot, summarize, search, and more) for AI coding assistants.

Improved:

  • Search orchestration alignment for accelerator usage: Updated setup and compose behavior for Video Search so NPU device selections are preserved and propagated consistently for DataPrep and multimodal embedding services.

  • Simplified per-component device model (Compose): Retired the redundant VDMS_DATAPREP_DEVICE baseline knob. Device selection is now purely per-component — DATAPREP_EMBEDDING_DEVICE, DATAPREP_DETECTION_DEVICE, and MME_EMBEDDING_DEVICE (each defaults to CPU) — matching the Helm chart model. ENABLE_EMBEDDING_GPU is now a mode-aware embedding shortcut (sdk→DataPrep embedding GPU, api→MME embedding GPU).

  • Helm accelerator support for search stack: Updated VSS Helm subcharts for multimodal-embedding-ms and multimodal-dataprep to support NPU as an accelerator path (device-key validation, resource requests/limits, and /dev/accel mounts).

  • Helm accelerator device permissions: Added global.accelGroupIds so the host gids owning /dev/dri (GPU) and /dev/accel (NPU) are injected into the pod supplementalGroups, letting the non-root container open the accelerator device (mirrors the Compose group_add render/video groups). Fixes NPU/GPU device initialization falling back to CPU-only.

  • Helm OpenVINO™ model cache: Added a persistent OpenVINO™ cache (ovCacheDir, default /app/ov_models/ov_cache) for multimodal-embedding-ms and multimodal-dataprep, plus a longer DataPrep startupProbe budget, so GPU/NPU model compilation completes once and is reused across pod restarts instead of recompiling (avoids startup crash loops).

  • Helm single-source image override: global.registry, global.tag, and global.pullPolicy now apply across all VSS service images (pipeline-manager, video-ingestion, video-search, vss-ui, multimodal-dataprep, multimodal-embedding-serving) from one place, with independent per-service PVCs for model/cache data.

  • Helm value override precedence: Subchart defaults no longer shadow global.env overrides for MM_DATAPREP_ALLOW_DUPLICATE_UPLOADS and SDK_USE_OPENVINO, so --set global.env.<KEY> now takes effect.

  • Clearer embedding error reporting: The video embedding flow now surfaces the real upstream DataPrep error instead of a misleading “Request timed out” message; only genuine timeouts (408/504/connection aborts) are reported as timeouts.

  • Graceful OVMS cache fallback: setup.sh now degrades cleanly when GPU drivers are absent instead of failing model cache setup.

  • Original filenames in embedding metadata: The uploaded video name is threaded end-to-end into stored embedding metadata, so search results show the original filename.

  • Service health and startup dependencies hardened across the stack, with retries for transient batch job polling failures.

  • Search deployment documentation refresh: Added a dedicated Deployment Options for Video Search matrix (SDK/API with CPU/GPU/NPU combinations), including explicit DATAPREP_EMBEDDING_DEVICE, MME_EMBEDDING_DEVICE, and DATAPREP_DETECTION_DEVICE examples for accelerator-specific routing.

  • Helm user-guide clarifications: Updated Helm guidance to include NPU device/key combinations and matching-device recommendations for shared PVC scheduling.

  • Documentation: Aligned user and agent documentation with the multi-backend architecture, refreshed the DataPrep, retriever, and VSS OpenAPI specs, added 4K/8K video ingestion troubleshooting guidance, and split 2025 release notes into a separate document.

  • Maintenance: Refreshed dependency version constraints and lock files, pinned Dockerfiles to release versions, aligned the DataPrep bucket with the video summary bucket, updated the VSS health check with a manual CI workflow trigger, and removed the profile_dataprep script and the vestigial EMBEDDING_PROCESSING_MODE chart value.

Fixed:

  • Duplicate-upload conflicts no longer leave orphaned video tiles in the UI.

  • Search result tiles no longer render as black thumbnails.

  • Assorted search-ms and video embedding flow defects.

Version 2026.1.0#

Release Date: June 17, 2026

New:

  • EXPERIMENTAL - vLLM Intel® Arc™ Pro B-series GPU Support: Added initial experimental support for running vLLM on Intel® Arc™ Pro B-series GPUs (XPUs) via new ENABLE_VLLM_GPU environment variable and docker/compose.vllm.xpu.yaml Docker Compose overlay. This feature enables GPU-accelerated VLM captioning and LLM summarization on Intel® Arc™ Pro B-series hardware (e.g., B60, B65, B70).

    Note

    This is an experimental feature in early stages and may require additional tuning and optimization. Performance characteristics are still being evaluated. Not recommended for production use.

  • Added a a new Dual UI mode with a new --summary --search CLI argument for setup.sh that allows running both the summary and the search applications simultaneously at /summary and /search URI endpoints respectively.

  • Added Dual UI support for Helm chart installations by allowing a values override file to be provided for summary and search modes simultaneously.

Improved:

  • Updated setup script and nginx configuration files to allow flexible UI routing for each existing mode of deployment (summary mode, search mode, Unified UI Mode) and the new Dual UI mode.

  • Refactored Helm chart to use a reusable vssui subchart with multi-mode nginx and consolidated embedding model config under global.embeddingModelName.

  • Updated DL Streamer base image to 2026.1.0-ubuntu24-rc1 for Video Ingestion Microservice.

  • Setup Script: Updated the environment variable to setup embedding models. New MULTIMODAL_EMBEDDING_MODEL and the existing TEXT_EMBEDDING_MODEL are used to provide embedding models in relevant modes.

  • Docker Compose: Replaced curl with Python urllib package in the container healthcheck command for a lighter runtime footprint for Audio Analyzer.

  • Docker Compose: Replaced environment variables with hard-coded mount paths. This helps in stopping containers without looking for preset variables.

  • Build Script: Removed Audio-Analyzer from the dependency build pipeline. A frozen version 1.3.3 will be used for the Audio Analyzer microservice for the current and all subsequent releases.

  • Setup Script: Removed unused environment variables and several environment variables being used as mount directories in Docker Compose files.

Version 1.3.3-rc1#

Release Date: 05 May 2026

Features:

  • Configurable final video summary: Added PM_PRODUCE_FINAL_SUMMARY feature flag to make the final LLM map-reduce video summary optional. When disabled, chunk-wise summaries are displayed chronologically instead. A per-video UI override checkbox is available in both upload flows. Audio transcript summarization is automatically skipped when the final summary is turned off.

  • Audio transcript summarization: Added audio transcript summarization support and improved audio transcription accuracy.

  • OVMS-first architecture: Replaced the standalone vlm-openvino-serving microservice with OpenVINO™ Model Server (OVMS) as the unified inference backend for both VLM captioning and LLM summarization. This is a breaking change; the vlm-inference subchart and container have been removed.

  • Performance Optimizations (MME & VDMS-Data-Prep):

    • Refactored pre-processing and inference with AsyncInferQueue based OpenVINO™ inference and static shape model compilation for iGPU.

    • Added ThreadPool for parallel open_clip image pre-processing with support for input tensor batching and padding for optimal OpenVINO™ inference paths.

    • Introduced PyAV-based video decode abstraction supporting keyframes and uniform sampled frames extraction with producer-consumer pattern for parallel decode and frame translation to PIL.

    • Enabled multiple/parallel decoder instances for file, RTSP stream, and bytes input sources.

    • Implemented frame batching for pipelined pre-processing and inference with integrated PyAV decoder in VDMS data-prep.

  • Search Timeout and Resource Management: Added SEARCH_DATAPREP_TIMEOUT_MS configuration to prevent VSS-UI timing out during embedding creation. Added ulimit constraints with soft and hard limits to enable shared memory creation and define memory block allocation boundaries.

HW used for validation:

  • Intel® Xeon® 5 + Intel® Arc™ B580 GPU

  • Vanilla Kubernetes Cluster

Known Issues/Limitations:

  • This release includes only limited testing on EMT‑S and EMT‑D, some behaviors may not yet be fully validated across all scenarios.

  • HW sizing of the Video Search or Video Summarization pipeline is in progress. Optimization of the pipelines will follow HW sizing.

  • Known issues are internally tracked. Reference not provided here.

  • how-to-performance document is not updated yet. HW sizing details will be added to this section shortly.

  • NPU support with OVMS is added as experimental feature and may not work for all models or configurations.

Version 1.3.2#

Release Date: 17 Feb 2026

Features:

  • In VSS search mode, users can now filter results by time range via:

  • Query parsing to infer time ranges (e.g., “person seen in last 5 minutes”).

  • Direct time range input from the UI.

  • Added live system and DataPrep performance metrics in the search UI (enable with export ENABLE_METRICS_MANAGER=true).

  • Fixed the build script of the vdms-dataprep microservice.

  • Added telemetry collection of the application metrics for VDMS-dataprep microservice and VLM microservice at /telemetry endpoint.

HW used for validation:

  • Intel® Xeon® 5 + Intel® Arc™ B580 GPU

  • Vanilla Kubernetes Cluster

Previous Releases#