Release Notes: Video Search and Summarization Sample Application#
Version 2026.3.0-rc1#
Release Date: November 6, 2026
Note: Helm Chart support will be deprecated in the next release. Limited validation has been performed on the Helm charts for this release.
New:
MCP server for AI agents: The VSS MCP tool server now starts with every Docker Compose deployment (
docker/compose.mcp.yaml) and stops withsetup.sh --stop; the Helm chart does not deploy it. The OpenAPI proxy was replaced with hand-declared tools that cover video listing, upload and indexing, text and image search, saved searches (create, poll, list, refetch and watch), summaries that follow the UI defaults unless the caller overrides them, live camera management, single-frame retrieval, tags and deployment info.Image search by reference: Upload a query image once with
POST /search/images(remove it withDELETE /search/images/{imageId}) and pass the returnedimageUrltoPOST /searchinstead of inline base64. Hits returned byGET /search/{queryId}include a playable link to the matching video; setPM_PUBLIC_BASE_URL(Helmpipelinemanager.env.PM_PUBLIC_BASE_URL) when the gateway is reached through another host name.Tags on live cameras: Tags set on a camera when it is added or updated are registered with Pipeline Manager, so they appear in
GET /tagsand in the search tag filters right away.Refresh cached images on deploy:
setup.sh --pull-always(orPULL_ALWAYS=true,make deploy PULL_ALWAYS=true) pulls newer images for moving tags such aslatestinstead of reusing a cached copy.Live RTSP stream ingestion: Register RTSP cameras from the new Live Streams panel in the search UI and ingest them continuously into the search index. Cameras can be listed, filtered by state or tag, paused, resumed, reconfigured, and removed, with live frame and embedding counts refreshed over the existing websocket. Backed by new Pipeline Manager
/streamsendpoints that proxy the Multimodal DataPrep live-stream API. Available in--searchand--dual(--summary --search) modes, but not in--summary-and-search/--unified, where the index holds text embeddings of summaries; enable or disable the UI withUI_LIVE_STREAMS_FEATUREand bound index growth withLIVE_RETENTION_HOURS(defaults to 24 hours). RTSP credentials are redacted from all API responses, logs, and search results.Minimum peak score filter: Search can drop weak matches at retrieval: frames scoring below
minPeakScore(0-1) are filtered in the Vector Retriever before aggregation, so every returned result’s best frame scores at least that value. Set it per search in the UI (off by default; starts at 0.2 when turned on), onPOST /search,POST /search/query,POST /search/{queryId}/refetchandPATCH /search/{queryId}/watch, or as the deployment default withSEARCH_MIN_PEAK_SCORE(default0, off). The value is saved with each query, so watched-query refreshes are gated by it. The MCPvss_search_videoandvss_get_searchtools accept it asmin_peak_score, and a watched search keeps the value saved on it.Sort and group search results by time: Pipeline Manager now dates every search result with
observedAtandtimeSource(live: camera capture time;upload: the video’s upload time plus the frame offset) and addsstreamIdto live hits. The search UI adds Sort by (Relevance, Newest first, Oldest first) and Group by (None, Tag, Camera / video) controls, replacing the Group by Tag button, and shows each result’s capture time, or its upload time with the match’s offset into the video (hits from one video sort by that offset). The MCPvss_search_videoandvss_get_searchtools acceptsort(relevance,newest,oldest) and returnobserved_at/time_sourceon every hit. Sorting reorders the top results by relevance; it does not fetch lower-ranked moments.Geo Spatial Map View: Map View in the search results header shows results as pins on a map beside the results, one per location, with its clip count. Off by default; enable it with
APP_MAP_FEATURE=FEATURE_ONand see Geo Spatial Map View.Re-add recent uploads: Several videos can be picked from the recent uploads list in the search UI; each is added again as a new video with the same name and the copies are embedded in one batch job. Backed by the new
POST /videos/{videoId}/copyendpoint, which copies the stored file to a newvideoIdinside the object store.
Improved:
Dependency refresh: Updated the Pipeline Manager and UI dependency sets to the latest in-range patch/minor releases. UI advisories drop from 6 to 0; Pipeline Manager drops from 19 to 8. The residual Pipeline Manager advisories come from three transitive packages, each unfixable without causing a regression:
braces(via the@swc/clibuild tool, dev-only) has no patched release — every published version is affected.stream-jsonanddecode-uri-componentarrive throughminio@8.0.7, the latest release.npm audit fix --force“fixes” them by downgrading minio to7.1.3, and minio only importsstream-json/jsonl/Parser.js, not the vulnerablepick/ignore/filter/replacefilters named in the advisory. Overridingdecode-uri-componentto the patched0.5.0is also not viable: that release is ESM-only and would make minio’s CommonJS build throwERR_REQUIRE_ESMat runtime.
Dependency override cleanup: Removed 9 now-redundant
overridesentries from Pipeline Manager (sha.js,qs,systeminformation,fast-xml-parser,socket.io-parser,lodash,body-parser,multer,serialize-javascript) and all 9 from the UI, after confirming each one no longer changes dependency resolution. The remaining Pipeline Manager overrides are retained because dropping them measurably readmits older duplicate copies (ajv,minimatch,picomatch,glob,webpack) or downgrades a package outright (diff8.0.4 → 4.0.4).Live camera limit is configurable:
LIVE_STREAM_MAX_CONCURRENT(default8; Helmmultimodaldataprep.liveStream.maxConcurrent) sets how many cameras ingest at once. Registering more returns503; pause or delete a camera to free a slot.Search filters panel: The time range, number of results and minimum peak score filters share one panel. Edits are drafts until Apply, and sliders show their value while dragging.
Result tags in every view: Each search result shows its tags with Group by set to None as well, and each tag gets a stable colour derived from its name, dark enough for white text.
OVMS 2026.4.1: Docker Compose and Helm now use OpenVINO™ Model Server
2026.4.1, and Pipeline Manager waits for a healthy OVMS whenever OVMS serves the VLM or LLM, so a deployment fails visibly instead of running without a model server. OVMSmax_num_seqsis sized from the Pipeline Manager concurrency (at least 16; override withOVMS_MAX_NUM_SEQS), andOVMS_CACHE_SIZE_GBis passed through. Pre-convertedOpenVINO/*models take their weight format from the repository name and are checked against the target device before download.NPU LLM prompt limits: An LLM on NPU is compiled with
OVMS_NPU_MAX_PROMPT_LEN(default8192) andOVMS_NPU_MIN_RESPONSE_LEN(default1024), and Pipeline Manager’s context and completion budgets are derived from them, so long summary prompts are no longer rejected by the NPU. A vision-language model set as the NPU LLM is rejected at setup.Inference concurrency defaults: With OVMS,
setup.shdefaults toPM_VLM_CONCURRENT=4andPM_LLM_CONCURRENT=2on every device (Helmpipelinemanager.env.VLM_CONCURRENT6,LLM_CONCURRENT2); vLLM still uses1.Faster startup detection: The Pipeline Manager container healthcheck polls every 5 seconds with a longer retry budget, so dependent services start sooner and a slow first start is not reported as unhealthy.
Platform updates: Video Ingestion uses the
intel/dlstreamer-pipeline-server:2026.3.0-20261027-weekly-ubuntu24base image, the summarizer CLI requires Go 1.26, and the Multimodal Embedding Serving and DataPrep images install the Intel® GPU compute runtime directly in the Dockerfile and fail the build if a driver is missing. Python, Node.js and Go dependencies were refreshed to clear fixable image vulnerabilities in VSS-owned code; remaining findings come from base images.The browser tab title no longer carries the “Intel EGAI” prefix.
DataPrep object detection uses Hugging Face OpenVINO models: Set
DATAPREP_DETECTION_MODEL(Helmmultimodaldataprep.frameProcessing.detectionModelName) to anOpenVINO/<repo>object-detection model, for exampleOpenVINO/yolox_s-int8-ov;OpenVINO/yolox_tiny-fp16-ov,OpenVINO/yolox_l-int8-ovandOpenVINO/YOLO11n-fp16-ovare also verified on CPU, GPU and NPU. It has no default and is required in--searchand--summary --searchmode unlessENABLE_OBJECT_DETECTION=false;--summary-and-searchmode always runs without detection. The optionalDATAPREP_DETECTION_CLASSES(HelmdetectionClasses) keeps only the listed classes. Models are cached in thedocker_dataprep-detection-modelsvolume; the olddocker_dataprep-yolox-modelsvolume is no longer used and can be removed.
Fixed:
Video embedding status is recorded:
GET /videosandGET /videos/{videoId}always returnedsearchEmbeddingsandtextEmbeddingsasnull, so clients, including the MCP server, could not tell whether a video was indexed. Pipeline Manager now records them from single and batch frame-embedding requests and from summary caption embedding; atrueflag is never downgraded. The unusedvideoEmbeddingsfield was removed. Live cameras have no video record and keep reporting progress throughGET /streams.Stalled ingestion no longer hangs: A DataPrep video ingestion whose detection or embedding inference never returns (for example after a GPU hang) now fails after
MM_DATAPREP_PIPELINE_STALL_TIMEOUT_Sseconds without progress (default300,0disables it; Helmmultimodaldataprep.sdkPipeline.pipelineStallTimeoutS), instead of staying in progress forever.Blank captions and summaries with reasoning models on OVMS: Reasoning-capable models could spend the whole completion budget thinking and return empty content. Pipeline Manager now disables thinking (
chat_template_kwargs.enable_thinking=false) on OVMS requests; models that do not use the flag are unaffected.make pushskipped the MCP image:make pushnow pushesvss-mcp-serverwith the same namemake builduses.
Upgrade Notes:
Breaking: In
--searchand--summary --searchmode,DATAPREP_DETECTION_MODELmust be set (orENABLE_OBJECT_DETECTION=false); GitHub YOLOX names such asyolox_sare no longer accepted. On Helm, setmultimodaldataprep.frameProcessing.detectionModelName.Each detection model keeps its own license (YOLOX is Apache-2.0, YOLO11 is AGPL-3.0); review the model card before use.
Building the summarizer CLI needs Go 1.26 or later.
Known Issues:
QwenText embedding crashes on Intel® Core™ Ultra processors (Nova Lake) on CPU:
multimodal-embedding-servingandmultimodal-dataprepcrash withSIGSEGVon the first inference when aQwenText/qwen3-embedding-*model (the default for--summary-and-searchmode) runs on CPU with OpenVINO™. Gunicorn logsWorker (pid:<N>) was sent SIGSEGV!and restarts the worker in a loop. This is an OpenVINO™ CPU plugin bug in releases up to 2026.4.1, fixed upstream for 2026.5. As a workaround, run embedding on CPU without OpenVINO™; see QwenText Embedding Crashes on Nova Lake CPUs.
Version 2026.2.0#
Release Date: September 9, 2026
New:
Backend-agnostic vector search: Select the vector database with
VECTORDB_BACKEND(vdmsdefault,milvus). All vector similarity search is delegated to the standalonevector-retrievermicroservice;video-searchkeeps query orchestration and frame-to-video aggregation, so results are identical in shape across backends.Multimodal DataPrep: Replaced the legacy VDMS DataPrep with the
multimodal-dataprepmicroservice across Docker Compose, Helm, and documentation.Metrics Manager integration: Replaced the legacy telemetry collector; the search UI now shows live multi-GPU telemetry (
ENABLE_METRICS_MANAGER=true).Batch video upload and embedding: Multi-select upload in the UI uploads all files and submits a single asynchronous batch embeddings job, with new
search-embeddings-batchandsearch-embeddings-jobsPipeline Manager endpoints and job-status polling. Single-file upload keeps the synchronous path.Directory watcher batch processing: The directory watcher service now batches video uploads and embedding creation.
Duplicate-content upload policy: New
MM_DATAPREP_ALLOW_DUPLICATE_UPLOADSsetting (Docker Compose and Helmglobal.env) rejects content-identical re-uploads with HTTP409. Enforced per item for batch-processed media, so a duplicate fails only its own item.Model download support: Models are now fetched by a dedicated model-download step in
setup.sh, including generic YOLO ids from the ultralytics hub for object detection.Search score transparency: Search results show the query-normalized relevance score alongside the underlying raw segment score and peak frame similarity, with a per-stage scoring breakdown popover and a weak-match indicator.
Makefile, functional test suite, and CI security scans for the Video Search and Summarization sample application.
AI agent skills: Added VSS agent skills (deploy, build, troubleshoot, summarize, search, and more) for AI coding assistants.
Improved:
Search orchestration alignment for accelerator usage: Updated setup and compose behavior for Video Search so NPU device selections are preserved and propagated consistently for DataPrep and multimodal embedding services.
Simplified per-component device model (Compose): Retired the redundant
VDMS_DATAPREP_DEVICEbaseline knob. Device selection is now purely per-component —DATAPREP_EMBEDDING_DEVICE,DATAPREP_DETECTION_DEVICE, andMME_EMBEDDING_DEVICE(each defaults toCPU) — matching the Helm chart model.ENABLE_EMBEDDING_GPUis now a mode-aware embedding shortcut (sdk→DataPrep embedding GPU,api→MME embedding GPU).Helm accelerator support for search stack: Updated VSS Helm subcharts for
multimodal-embedding-msandmultimodal-dataprepto support NPU as an accelerator path (device-key validation, resource requests/limits, and/dev/accelmounts).Helm accelerator device permissions: Added
global.accelGroupIdsso the host gids owning/dev/dri(GPU) and/dev/accel(NPU) are injected into the podsupplementalGroups, letting the non-root container open the accelerator device (mirrors the Composegroup_addrender/video groups). Fixes NPU/GPU device initialization falling back to CPU-only.Helm OpenVINO™ model cache: Added a persistent OpenVINO™ cache (
ovCacheDir, default/app/ov_models/ov_cache) formultimodal-embedding-msandmultimodal-dataprep, plus a longer DataPrepstartupProbebudget, so GPU/NPU model compilation completes once and is reused across pod restarts instead of recompiling (avoids startup crash loops).Helm single-source image override:
global.registry,global.tag, andglobal.pullPolicynow apply across all VSS service images (pipeline-manager, video-ingestion, video-search, vss-ui, multimodal-dataprep, multimodal-embedding-serving) from one place, with independent per-service PVCs for model/cache data.Helm value override precedence: Subchart defaults no longer shadow
global.envoverrides forMM_DATAPREP_ALLOW_DUPLICATE_UPLOADSandSDK_USE_OPENVINO, so--set global.env.<KEY>now takes effect.Clearer embedding error reporting: The video embedding flow now surfaces the real upstream DataPrep error instead of a misleading “Request timed out” message; only genuine timeouts (
408/504/connection aborts) are reported as timeouts.Graceful OVMS cache fallback:
setup.shnow degrades cleanly when GPU drivers are absent instead of failing model cache setup.Original filenames in embedding metadata: The uploaded video name is threaded end-to-end into stored embedding metadata, so search results show the original filename.
Service health and startup dependencies hardened across the stack, with retries for transient batch job polling failures.
Search deployment documentation refresh: Added a dedicated Deployment Options for Video Search matrix (SDK/API with CPU/GPU/NPU combinations), including explicit
DATAPREP_EMBEDDING_DEVICE,MME_EMBEDDING_DEVICE, andDATAPREP_DETECTION_DEVICEexamples for accelerator-specific routing.Helm user-guide clarifications: Updated Helm guidance to include NPU device/key combinations and matching-device recommendations for shared PVC scheduling.
Documentation: Aligned user and agent documentation with the multi-backend architecture, refreshed the DataPrep, retriever, and VSS OpenAPI specs, added 4K/8K video ingestion troubleshooting guidance, and split 2025 release notes into a separate document.
Maintenance: Refreshed dependency version constraints and lock files, pinned Dockerfiles to release versions, aligned the DataPrep bucket with the video summary bucket, updated the VSS health check with a manual CI workflow trigger, and removed the
profile_dataprepscript and the vestigialEMBEDDING_PROCESSING_MODEchart value.
Fixed:
Duplicate-upload conflicts no longer leave orphaned video tiles in the UI.
Search result tiles no longer render as black thumbnails.
Assorted
search-msand video embedding flow defects.
Version 2026.1.0#
Release Date: June 17, 2026
New:
EXPERIMENTAL - vLLM Intel® Arc™ Pro B-series GPU Support: Added initial experimental support for running vLLM on Intel® Arc™ Pro B-series GPUs (XPUs) via new
ENABLE_VLLM_GPUenvironment variable anddocker/compose.vllm.xpu.yamlDocker Compose overlay. This feature enables GPU-accelerated VLM captioning and LLM summarization on Intel® Arc™ Pro B-series hardware (e.g., B60, B65, B70).Note
This is an experimental feature in early stages and may require additional tuning and optimization. Performance characteristics are still being evaluated. Not recommended for production use.
Added a a new Dual UI mode with a new
--summary --searchCLI argument forsetup.shthat allows running both the summary and the search applications simultaneously at /summary and /search URI endpoints respectively.Added Dual UI support for Helm chart installations by allowing a values override file to be provided for summary and search modes simultaneously.
Improved:
Updated setup script and nginx configuration files to allow flexible UI routing for each existing mode of deployment (summary mode, search mode, Unified UI Mode) and the new Dual UI mode.
Refactored Helm chart to use a reusable
vssuisubchart with multi-mode nginx and consolidated embedding model config underglobal.embeddingModelName.Updated DL Streamer base image to
2026.1.0-ubuntu24-rc1for Video Ingestion Microservice.Setup Script: Updated the environment variable to setup embedding models. New
MULTIMODAL_EMBEDDING_MODELand the existingTEXT_EMBEDDING_MODELare used to provide embedding models in relevant modes.Docker Compose: Replaced
curlwith Pythonurllibpackage in the containerhealthcheckcommand for a lighter runtime footprint for Audio Analyzer.Docker Compose: Replaced environment variables with hard-coded mount paths. This helps in stopping containers without looking for preset variables.
Build Script: Removed Audio-Analyzer from the dependency build pipeline. A frozen version 1.3.3 will be used for the Audio Analyzer microservice for the current and all subsequent releases.
Setup Script: Removed unused environment variables and several environment variables being used as mount directories in Docker Compose files.
Version 1.3.3-rc1#
Release Date: 05 May 2026
Features:
Configurable final video summary: Added
PM_PRODUCE_FINAL_SUMMARYfeature flag to make the final LLM map-reduce video summary optional. When disabled, chunk-wise summaries are displayed chronologically instead. A per-video UI override checkbox is available in both upload flows. Audio transcript summarization is automatically skipped when the final summary is turned off.Audio transcript summarization: Added audio transcript summarization support and improved audio transcription accuracy.
OVMS-first architecture: Replaced the standalone
vlm-openvino-servingmicroservice with OpenVINO™ Model Server (OVMS) as the unified inference backend for both VLM captioning and LLM summarization. This is a breaking change; thevlm-inferencesubchart and container have been removed.Performance Optimizations (MME & VDMS-Data-Prep):
Refactored pre-processing and inference with
AsyncInferQueuebased OpenVINO™ inference and static shape model compilation for iGPU.Added ThreadPool for parallel open_clip image pre-processing with support for input tensor batching and padding for optimal OpenVINO™ inference paths.
Introduced PyAV-based video decode abstraction supporting keyframes and uniform sampled frames extraction with producer-consumer pattern for parallel decode and frame translation to PIL.
Enabled multiple/parallel decoder instances for file, RTSP stream, and bytes input sources.
Implemented frame batching for pipelined pre-processing and inference with integrated PyAV decoder in VDMS data-prep.
Search Timeout and Resource Management: Added
SEARCH_DATAPREP_TIMEOUT_MSconfiguration to prevent VSS-UI timing out during embedding creation. Added ulimit constraints with soft and hard limits to enable shared memory creation and define memory block allocation boundaries.
HW used for validation:
Intel® Xeon® 5 + Intel® Arc™ B580 GPU
Vanilla Kubernetes Cluster
Known Issues/Limitations:
This release includes only limited testing on EMT‑S and EMT‑D, some behaviors may not yet be fully validated across all scenarios.
HW sizing of the Video Search or Video Summarization pipeline is in progress. Optimization of the pipelines will follow HW sizing.
Known issues are internally tracked. Reference not provided here.
how-to-performancedocument is not updated yet. HW sizing details will be added to this section shortly.NPU support with OVMS is added as experimental feature and may not work for all models or configurations.
Version 1.3.2#
Release Date: 17 Feb 2026
Features:
In VSS search mode, users can now filter results by time range via:
Query parsing to infer time ranges (e.g., “person seen in last 5 minutes”).
Direct time range input from the UI.
Added live system and DataPrep performance metrics in the search UI (enable with
export ENABLE_METRICS_MANAGER=true).Fixed the build script of the
vdms-dataprepmicroservice.Added telemetry collection of the application metrics for VDMS-dataprep microservice and VLM microservice at
/telemetryendpoint.
HW used for validation:
Intel® Xeon® 5 + Intel® Arc™ B580 GPU
Vanilla Kubernetes Cluster