# gvamotiondetect Performs lightweight motion detection on NV12 video frames and emits motion regions of interest (ROIs) as analytics metadata. Automatically uses a GPU accelerated path when GPU memory caps are negotiated (VAMemory on Linux, D3D11Memory or D3D12Memory on Windows); otherwise falls back to a system-memory (CPU) path. Designed for low-latency scene motion highlighting and downstream triggering without requiring a full inference model. ## Windows On Windows, the element accepts and produces: - `video/x-raw(memory:D3D11Memory)` in `NV12` - `video/x-raw(memory:D3D12Memory)` in `NV12` - `video/x-raw` in `NV12`, `I420` or `YV12` The properties are the same as on Linux. The GPU path depends on the negotiated memory type: - **D3D11Memory**: the frame is downscaled on the GPU with the D3D11 video processor, then converted to grayscale through OpenCV OpenCL/D3D11 interop. The frame is not copied to system memory. - **D3D12Memory**: the frame is downscaled on the GPU with the D3D12 converter. The small downscaled luma plane is then read back to system memory, because OpenCV has no D3D12 interop. The readback is cheap because the working resolution is small. The element obtains the upstream D3D11/D3D12 device through the GStreamer device context. If no device is available or a GPU step fails, it falls back to the system-memory path for that frame. Example pipelines: ```bash # D3D11 gst-launch-1.0 filesrc location=${VIDEO_FILE} ! parsebin ! d3d11h264dec ! gvamotiondetect ! queue ! gvametaconvert ! gvametapublish file-path=motion.json ! fakesink # D3D12 gst-launch-1.0 filesrc location=${VIDEO_FILE} ! parsebin ! d3d12h264dec ! gvamotiondetect ! queue ! gvametaconvert ! gvametapublish file-path=motion.json ! fakesink ``` ## Element Details (Linux) ```bash Pad Templates: SINK template: 'sink' Availability: Always Capabilities: video/x-raw(memory:VAMemory) format: (string)NV12 width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] video/x-raw format: (string)NV12 width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] SRC template: 'src' Availability: Always Capabilities: video/x-raw(memory:VAMemory) format: (string)NV12 width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] video/x-raw format: (string)NV12 width: [ 1, 2147483647 ] height: [ 1, 2147483647 ] framerate: [ 0/1, 2147483647/1 ] Element has no clocking capabilities. Element has no URI handling capabilities. Pads: SINK: 'sink' Pad Template: 'sink' SRC: 'src' Pad Template: 'src' Element Properties: block-size : Full-resolution block size (pixels) used to build the grid for per-block motion ratio. flags: readable, writable Integer. Default: 64 (range 16..512) motion-threshold : Per-block changed pixel ratio (0..1) required to flag motion in that block before temporal confirmation. flags: readable, writable Double. Default: 0.05 min-persistence : Frames a tracked ROI must persist (age) before being eligible for emission. flags: readable, writable Integer. Default: 2 (range 1..30) max-miss : Grace frames allowed after last successful match before a tracked ROI is purged. flags: readable, writable Integer. Default: 1 (range 0..30) iou-threshold : Intersection-over-Union threshold (0..1) used to match raw motion rectangles to existing tracked ROIs. flags: readable, writable Double. Default: 0.3 smooth-alpha : Exponential moving average smoothing factor (0..1) for ROI coordinates; higher values follow motion faster. flags: readable, writable Double. Default: 0.5 confirm-frames : Consecutive frames required to confirm a motion block. 1 = immediate single-frame detection (most sensitive). flags: readable, writable Integer. Default: 1 (range 1..10) pixel-diff-threshold : Per-pixel absolute luma difference (1..255) applied before thresholding; lower values increase sensitivity. flags: readable, writable Integer. Default: 15 min-rel-area : Minimum relative frame area (0..0.25) a motion rectangle must cover to be considered (filters tiny noise boxes). flags: readable, writable Double. Default: 0.0005 name : The name of the element instance. flags: readable, writable String. Default: "gvamotiondetect0" parent : The parent object. flags: readable, writable Object of type "GstObject" ``` ## Relationship between confirm-frames and min-persistence These two controls act at different stages and are not interchangeable: - confirm-frames works at the raw per-block motion stage. It requires N consecutive frames of activity in the same grid block before that block contributes to a raw motion rectangle. It filters out flicker/noise early. - min-persistence applies after rectangles are merged and tracked. A tracked ROI must be matched for N frames (its age) before it is emitted. It guards publication of unstable, short-lived motion regions. Effect: Raising confirm-frames reduces how many raw rectangles enter tracking (front-end suppression). Raising min-persistence delays emission of tracked ROIs (back-end stabilization). You can combine them: e.g., confirm-frames=2 with min-persistence=2 yields rectangles only after two agreeing frames and then requires two tracked frames for output (at least 2 total, possibly 3 if initial confirmation overlaps). Keeping confirm-frames=1 but min-persistence>1 allows immediate raw detection yet still waits for persistence before emission. ## Metadata Output - Each emitted motion ROI is attached as a `GstVideoRegionOfInterestMeta` with label "motion". - A `GstAnalyticsRelationMeta` aggregates object detection metadata entries (type quark "motion") for all ROIs on the frame. - ROI coordinates are normalized internally for analytics structures and rounded to 3 decimal places to reduce payload size. ## Algorithm Summary 1. Acquire current frame luma plane (VA surface, D3D11 or D3D12 GPU path, or system-memory conversion) and downscale to a working size. 2. Build motion mask: absdiff(previous, current) -> GaussianBlur -> threshold (pixel-diff-threshold) -> morphology (open + dilate). 3. Grid scan: accumulate changed pixels per block, compute ratio; mark blocks exceeding motion-threshold. 4. Temporal confirmation: if confirm-frames > 1 require consecutive active frames per block before rectangle creation. 5. Merge overlapping rectangles, track over time with IoU matching and exponential smoothing (smooth-alpha). 6. Apply persistence (min-persistence) and miss grace (max-miss), then emit stable ROIs with associated analytics metadata. ## Tuning Guidelines - Increase `pixel-diff-threshold` to reduce noise sensitivity (e.g., minor lighting flicker). Decrease for subtle motion. - Increase `motion-threshold` to require more changed pixels in a block (filters small localized changes). - Raise `confirm-frames` (e.g., 2-3) to suppress transient single-frame spikes; keep at 1 for maximal responsiveness. - Adjust `min-persistence` if you want to delay ROI publication until sustained movement is observed. - Lower `iou-threshold` if objects move rapidly and fail to match across frames; raise to avoid merging nearby independent motions. - Set `smooth-alpha` closer to 1.0 for minimal smoothing (snappier boxes) or lower for steadier boxes. - Raise `min-rel-area` to suppress very small (potentially noisy) motion rectangles; lower it to allow detection of tiny/distant objects. Default 0.0005 ≈ 0.05% of frame area. ## Performance Notes - VA-API path uses hardware surface mapping and optional hardware downscale (via `vaBlitSurface` when available) to minimize memory bandwidth. - D3D11 path downscales on the GPU and keeps the frame on the GPU through OpenCV OpenCL/D3D11 interop. - D3D12 path downscales on the GPU and reads back only the small luma plane. - System-memory path performs software resize; consider reducing `block-size` to balance granularity vs CPU cost. - Internal working resolution scales proportionally to input width when downscaling; large frames benefit more from confirmation and smoothing. ## Limitations / Future Improvements - Only NV12 format is currently supported (both system memory and VAMemory). On Windows, system memory also accepts I420 and YV12, while D3D11Memory and D3D12Memory accept NV12 only. - Block merging uses a simple O(n^2) combination; extremely dense motion may produce fewer, larger ROIs. - No explicit per-ROI confidence beyond binary motion presence (confidence fixed to 1.0 in analytics metadata). - Coordinate rounding (3 decimal places) applied for analytics metadata; raw ROI meta stores integer pixel coordinates. ## Plugin Registration ```text Plugin Registration: Name: gvamotiondetect Classification: Filter/Video Description: Automatically uses VA surface path when VAMemory caps negotiated; otherwise system memory path ``` On Windows, the description is "Windows motion detection: D3D11/D3D12 GPU path when negotiated, software otherwise".