Release Notes: Multimodal Data Preparation for Retrieval#
Version 2026.3.0-rc1#
Release Date: November 6, 2026
New:
On-demand single-frame retrieval (
GET /media/frame): reconstruct one frame from already-stored media (an uploaded file or a livesegments/*.mp4clip) and return it as rawimage/jpegor as base64 inside JSON. Frames are decoded on demand and never written back to storage. The endpoint is query-agnostic — it addresses a frame byvideo_id/stream_id+timestamp(+ an optionalcrop_bboxforvariant=crop) — so a downstream agent can take the address straight from asearch-msresult’sbest_frame_infoand fetch the peak-scoring frame to hand to a VLM. In VSS this is also exposed through pipeline-manager atGET /manager/frames.Live (RTSP) stream ingestion as a managed resource: the new
/media/streamsCRUD surface registers a camera and ingests it continuously on a background worker — create (single and batch), list withstate/tagsfilters, inspect stats, pause/resume and reconfigure viaPATCH, and delete (single and batch) with optionalpurge_embeddings/purge_media. Registrations are persisted and restored on restart, dropped sources reconnect with a bounded budget, and live embeddings are addressable through the existing media endpoints (bucket_name=MM_DATAPREP_LIVE_STREAM_BUCKET,video_id=stream_id).Playback media for live streams: each stream records N-second MP4 segments (
MM_DATAPREP_LIVE_SEGMENT_DURATION_SECONDS, default 10), and every live embedding points at the segment covering its frame — so a retrieval hit on a live camera is playable.Live retention:
MM_DATAPREP_LIVE_RETENTION_HOURS(default0= keep forever) prunes live embeddings and media past the window; requires the new vector-storedelete_embeddings_before()operation, implemented for both VDMS and Milvus.Live telemetry and health: live sessions are recorded by
GET /telemetry, andGET /healthreports live-stream counts per state.External media correlation contract: every live embedding now records
sensor_id(stable identity of the physical source, settable at registration and defaulting to thestream_id),capture_timewith an explicitcapture_time_source, andmedia_owner. These let media stored by another service be matched back to an embedding without changing the decode or embedding path. See Stream Manager Integration Readiness.Camera-clock capture timestamps for live streams:
capture_timeis derived from each frame’s PTS anchored to the best available clock, rather than from the ingest instant. This removes per-frame pipeline jitter and lets an independent consumer of the same RTSP feed resolve the same frame. The source degrades throughrtcp_sender_report→stream_anchored→ingest_estimated, reported incapture_time_source; implausible camera clocks are rejected rather than trusted. Segment bucketing is unchanged and still uses ingest time. In practice expectstream_anchored: validation against a Hikvision DS-2CD1023G0E-I found the camera emits no RTCP at all — a raw probe saw 4316 RTP packets and 0 RTCP packets over 45 s on TCP, and none on UDP — so no client can recover an NTP mapping. Always branch oncapture_time_sourcerather than assuming camera-clock accuracy.Camera clock-skew warning: starting a live stream probes the device’s HTTP
Dateheader and logs a warning when its clock disagrees with the host by more thanMM_DATAPREP_LIVE_CLOCK_SKEW_WARN_SECONDS(default 2). The check is vendor-neutral, sends no credentials, runs off the ingestion path, and never alters a stored timestamp — it surfaces the otherwise-invisible condition that makes a camera unusable as a shared time reference when correlating with an external recorder. It distinguishes genuine drift from embedded web servers that stamp local time but label itGMT. Disable withMM_DATAPREP_LIVE_CLOCK_CHECK_ENABLED=false.Object detection with Hugging Face OpenVINO models: set
MM_DATAPREP_DETECTION_MODEL_NAMEto any object-detection model in the Hugging FaceOpenVINOnamespace supported by OpenVINO Model API, for exampleOpenVINO/yolox_s-int8-ov. Input size, NMS and class labels come from the model’s own metadata, and only the selected model is downloaded on first use intoMM_DATAPREP_DETECTION_MODEL_DIR. Detection runs on CPU, GPU or NPU;OpenVINO/yolox_s-int8-ov,OpenVINO/yolox_tiny-fp16-ov,OpenVINO/yolox_l-int8-ovandOpenVINO/YOLO11n-fp16-ovare verified on all three.Detection class filter:
MM_DATAPREP_DETECTION_CLASSES(for exampleperson,car) keeps only the listed classes. Names are checked against the model’s labels at startup.GET /healthreportsdetection_model,detection_deviceanddetection_classes.
Improved:
GPU driver install fails loudly: the Dockerfile installs the Intel® GPU compute runtime
26.35.39758.10(IGC2.41.5) directly instead of throughinstall_ubuntu_gpu_drivers.sh, retries downloads, and fails the build if the Level Zero GPU driver or OpenCL ICD is missing. Versions are build arguments (GPU_COMPUTE_RUNTIME_VERSION,GPU_IGC_VERSION,GPU_IGC_BUILD,GPU_GMMLIB_VERSION).Refreshed Python dependencies within the existing constraints to clear fixable image vulnerabilities.
Fixed:
Live streams now produce embeddings promptly. Batches were only flushed once
MM_DATAPREP_VIDEO_EXTRACTION_BATCH_SIZEsampled frames had accumulated, and the “drain whatever is left” path only runs at end-of-stream — which an RTSP source never reaches. A camera at 12.5 fps withframe_interval=15therefore produced nothing for roughly five minutes while pinning shared-memory blocks. Partially filled batches from live sources are now flushed once the oldest frame reachesMM_DATAPREP_LIVE_BATCH_MAX_AGE_SECONDS(default 20).Live stream statistics no longer stay at zero.
frames_processedandembeddings_createdwere only updated after the ingestion pipeline returned, which for a continuous stream never happens. Progress is now reported per stored batch, soGET /media/streams/{stream_id}reflects ingestion while it runs.Recorded live media is now downloadable.
GET /medialists live objects under<stream_id>/segments/..., butGET /media/downloadrejected the/in those paths and a barevideo_idreturned an arbitrary object — so a retrieval hit on a live camera could not be played back. The endpoint accepts a newmedia_pathquery parameter resolved strictly within the stream’s own prefix.External correlation fields now reach the vector database.
sensor_id,capture_time,capture_time_source, andmedia_ownerwere computed per frame but were not part of the enforced canonical metadata contract, so they were stripped before storage and never appeared in retrieval results.Live sessions now appear in
GET /telemetry. Records were built with a-1“unknown” stream index that failed the non-negative schema constraint, so every live record was silently discarded; the counts were also reported as zero because the live pipeline does not surface totals at the top level of its result.Camera credentials no longer reach the service logs. The decoder recorded the raw RTSP URL as the stream’s source in the metadata it logs verbatim, disclosing any embedded
user:pass@. That value is now redacted with the same helper used for API responses and vector metadata.Live ingestion no longer stores embeddings under a placeholder identity (
RTSP_BUCKET/video_id=-1), which made them impossible to list, filter, or delete.Ingestion no longer hangs on a stuck inference call. If a detection or embedding inference never returned (for example after a GPU hang), the video job stayed
runningforever. A stall watchdog now fails the job when no pipeline stage makes progress forMM_DATAPREP_PIPELINE_STALL_TIMEOUT_Sseconds (default300,0disables it), with an error that names the stalled stage. See Troubleshooting.Ingestion no longer hangs when a decoder thread dies. A frame larger than the shared-memory block size (for example 4K portrait video against the 1920x1080 default) killed the producer thread without signalling completion, leaving the request spinning forever and never returning. Decoder failures are now forwarded to the consumer and raised, and the request is rejected with
400 Bad RequestnamingMM_DATAPREP_VIDEO_SHM_BLOCK_SIZEand the required byte count instead of an opaque500. The failure path also shuts down the shared-memory pools, which previously leaked on every failed request.Concurrent embedding requests no longer abort the worker. Two overlapping inference calls could overwrite each other’s async completion callback, writing a result into a wrong-sized buffer and terminating the process with SIGABRT. Inference submission is now serialized per model instance and callback exceptions are re-raised in the submitting thread.
ROI consolidation metadata is no longer shared between requests.
merged_boxes_countandcontext_expansion_appliedwere held on the shared detector, so overlapping requests could attach another frame’s values to a crop. They are now kept per thread.ROI consolidation honours
0.MM_DATAPREP_ROI_CONSOLIDATION_IOU_THRESHOLD=0orMM_DATAPREP_ROI_CONSOLIDATION_CONTEXT_SCALE=0was silently replaced with0.2.
Upgrade Notes:
Breaking:
POST /media/rtsphas been removed. Register the source withPOST /media/streamsinstead; the call returns immediately with astream_idrather than holding the request open for the life of the stream, andDELETE /media/streams/{stream_id}replaces client-disconnect as the stop signal.Live-stream registrations are stored in a PostgreSQL
live_streamstable (MM_DATAPREP_LIVE_STREAM_DB_*). The standalone compose bundles apostgresservice; in the VSS stack the registry reuses the shared PostgreSQL instance.Live ingestion grows the vector index indefinitely unless
MM_DATAPREP_LIVE_RETENTION_HOURSis set.Breaking:
MM_DATAPREP_DETECTION_MODEL_NAMEhas no default and is required while object detection is enabled. It takes anOpenVINO/<repo>Hugging Face id; GitHub YOLOX names such asyolox_sare rejected. SetMM_DATAPREP_ENABLE_OBJECT_DETECTION=falseto run without detection.Breaking:
src/config.yamlandCONFIG_FILEPATHhave been removed; all configuration is read fromMM_DATAPREP_*environment variables.Breaking: the
detection_confidencerequest parameter has been removed from the upload, process, image, batch, directory and live-stream APIs, and live-stream responses no longer return it. The value was never applied per request; the threshold is the service settingMM_DATAPREP_DETECTION_CONFIDENCE. Requests that still send the field are accepted and the field is ignored.MM_DATAPREP_DETECTION_MODEL_DIRnow defaults to/app/models/object-detection(was/app/models/yolox), and the Compose model volume isdataprep-detection-models(DETECTION_MODELS_VOLUME_NAME,DETECTION_MODELS_MOUNT_PATH). The olddataprep-yolox-modelsvolume is no longer used and can be removed.Each detection model keeps its own license (for example YOLOX is Apache-2.0, YOLO11 is AGPL-3.0); review the model card before use.
Version 2026.2.0#
Release Date: September 9, 2026
New:
Multimodal ingestion: the service now ingests images alongside video. Images are embedded directly (no frame extraction) into the same shared vector space as video frames and text summaries, discriminated by a
content_type(video/image/text) metadata field, enabling cross-modal search.Three image transports: multipart binary (
POST /media/upload), inline base64 and remote URL (POST /media/ingest, typed on atypediscriminator; batch viaPOST /media/ingest/batch).Async batch ingestion:
POST /media/upload/batch,/media/ingest/batch,/media/process/batch, and/media/ingest-dirreturn202 Acceptedwith ajob_idpolled atGET /media/jobs/{job_id}(cancellable viaDELETE). Per-item error isolation keeps one bad item from failing the whole job.Content deduplication: optional content-hash (SHA-256) dedup gated by
MM_DATAPREP_ALLOW_DUPLICATE_UPLOADS(defaulttrue); byte-identical re-uploads are rejected409 Conflictacross all transports.HTTP Range / seek support on
GET /media/download(206 Partial Content).Complete delete CRUD:
DELETE /media/{bucket}/{video_id}now removes both the stored object and its embeddings from the vector database.Ingest by reference:
store_copy=falseindexes media already present on a mounted path without copying bytes into object storage, using a canonical, path-traversal-safe metadata contract (MM_DATAPREP_INGEST_DATA_ROOT/INGEST_DATA_ROOT_HOST).RTSP source support in the embedding pipeline (
POST /media/rtsp).Metrics Manager integration: ingestion throughput is published for live observability.
Added expanded NPU device support in setup/runtime configuration for per-component execution (
MM_DATAPREP_EMBEDDING_DEVICE,MM_DATAPREP_DETECTION_DEVICE).Added richer API/OpenAPI alignment updates for media processing and management endpoints.
Improved:
Endpoints renamed
/videos/*→/media/*to reflect multimodal functionality (for example/videos/upload→/media/upload,/videos/minio→/media/process,/videos/batch/{job_id}→/media/jobs/{job_id}). Request/response field names (video_id,video_name,video_url) are unchanged for retriever compatibility.Backend-agnostic: vector database (
vdms/milvus) and object storage (minio/local) are each selected at startup behind a factory viaMM_DATAPREP_VECTORDB_BACKEND/MM_DATAPREP_STORAGE_BACKEND— no code changes to switch. See Pluggable Backends.Registry-based factories: vector-store and storage backends self-register via a decorator, so adding a backend is a single self-contained module with no factory edits.
Microservice renamed from
vdms-datapreptomultimodal-dataprep, removing VDMS-specific naming from generic identifiers.Environment variables normalized under a single
MM_DATAPREP_prefix, with fully independent per-component device selection.Single in-process embedding pipeline: the deprecated API embedding mode and the standalone multimodal-embedding-serving container were removed; embeddings are generated through the in-process Python SDK.
Object detection now applies to both video frames and images via the shared
MM_DATAPREP_ENABLE_OBJECT_DETECTIONtoggle.Hardened NPU runtime dependency installation in Docker images (including stricter Level Zero/driver setup validation).
Simplified containerization flow by removing legacy dev/lint/report runtime paths and aligning setup scripts with a production-focused image flow.
Updated compose/setup defaults and docs to reflect current accelerator-oriented configuration behavior.
Fixed:
Resolved a shared-memory pool deadlock: pool acquisition is now time-bounded and batch size is clamped to the pool capacity.
Fixed file-descriptor exhaustion (
[Errno 24] Too many open files) in the shared-memory pool: each block held two open descriptors, so the frame and detected-crop pools together needed roughly 3000 and failed under a 1024-descriptor limit. Blocks now release their descriptor after allocation.Fixed shared-memory segments leaking into
/dev/shmwhen pool teardown hit an already-unlinked block; blocks are now unlinked independently.Video processing is offloaded to a worker thread so long ingestions no longer block the event loop and stall
/health.Duplicate-upload policy is now enforced per item for batch-processed media (
POST /media/process/batch), matching the single-media path.Duplicate-upload conflicts no longer leave orphan tiles behind.
DataPrep object bucket aligned with the video summary flow.
Fixed an end-of-stream hang on the RTSP ingestion path.
Fixed Milvus connection failures on existing collections, plus Milvus compose environment wiring and healthcheck.
Fixed request-schema compatibility issue in upload processing parameters for newer FastAPI/Pydantic combinations.
Upgrade Notes:
Consumers of the old
/videos/*paths must migrate to/media/*.Environment variables not already prefixed with
MM_DATAPREP_must be renamed (for exampleMM_EMBEDDING_DEVICE→MM_DATAPREP_EMBEDDING_DEVICE).
Version 2026.1.0#
Release Date: June 17, 2026
New:
Stage-separated embedding pipeline: decode → detect → embed → store stages run concurrently via bounded queues with back-pressure control.
Shared memory Zero-copy frame metadata transport via POSIX shared memory pool between pipeline stages.
Pipeline tracer that emits Chrome Tracing JSON for profiling decode/detect/embed/store stages; enabled via
MM_DATAPREP_ENABLE_TRACING=true.Structured per-stream pipeline metrics: stage durations, throughput FPS, concurrency factor, and efficiency %. Runtime stats can be saved as JSON via
MM_DATAPREP_SAVE_RUNTIME_PIPELINE_STATS=true.Configurable embedding pipeline via environment variables (seeded by
setup.sh).
Improved:
Uploaded video bytes are processed directly from memory; no temp-file re-read after MinIO upload.
Batch embedding generation supports
metrics_out=Trueto return inference timing alongside results.Telemetry log now emits a structured pipeline summary (frames, detections, embeddings, FPS, stage durations) on completion.
Container healthcheck, raised
nofileulimits andipc: hostadded to Docker Compose.get-started.mdupdated with full environment variable reference and setup instructions.
Upgrade Notes:
Telemetry schema:
TelemetryRecord.stagesand.throughputreplaced bypipeline_stats,stage_duration, andstage_throughputdicts.batch_indexis now 0-based;stream_idfield added toTelemetryBatchDetailandTelemetryCounts. Update downstream telemetry consumers.Docker / Kubernetes deployments must set
ipc: host/hostIPC: truefor the shared memory pipeline.
Validated configuration:
Intel® Xeon® 5 + Intel® Arc™ B580 GPU, Intel® Core™ Ultra Processors (Series 2 and 3)
Vanilla Kubernetes Cluster
Releases 1.2.0, 1.2.1, 1.2.2, 1.2.3, 1.3.0 and 1.3.1#
This microservice supports features based on the requirements of Video Search and Summarization sample application which is using this microservice. Refer to Video Search and Summarization release notes for release details of this microservice.