How It Works#

The VMS Adapter Plugin (VAP) is a modular orchestration service that routes video streams from supported Video Management System (VMS) providers to AI Analytics Applications and relays results back to the provider dashboard or VMS.

Architecture#

VAP Analytics Integration architecture

Data Flow#

Camera Discovery#

  1. Operator triggers Discover Cameras from the dashboard or POST /v1/cameras/discover.

  2. The Orchestrator calls each registered VMS shim:

    • NxWitnessVmsShim queries Nx Witness GET /rest/v4/devices for all camera devices.

  3. The system persists discovered cameras to PostgreSQL with vendor-prefixed identifiers (IDs), such as nx:abc123-uuid.

  4. The dashboard displays the full camera list. Operators enable specific cameras for analytics.

Live Video Captioning (LVC) Flow#

Provider dashboard
    │  POST /v1/analytics-apps/live_captioning/runs  { camera_id, prompt, model, … }
    ▼
FastAPI route (analytics_apps.py)
    │  IAnalyticsAppShim.start(params)
    ▼
LiveCaptioningAnalyticsAppShim
    │  resolves camera_id → RTSP URL via NxWitnessVmsShim
    │  POST /api/runs  →  LVC backend (FastAPI)
    ▼
LVC DL Streamer Pipeline Server
    │  processes RTSP stream at configured frame rate
    ├─► VLM inference → captions → MQTT broker → LVC SSE stream
    └─► preview frames → MediaMTX (WebRTC)
    ▼
VAP  GET /v1/analytics-apps/live_captioning/results/stream  (SSE proxy)
    ▼
Provider dashboard
    │  caption overlay on WebRTC video player

DL Streamer Vision (e.g., Loitering Detection) Flow#

Provider dashboard
    │  POST /v1/analytics-apps/dls_vision/runs  { camera_id, pipeline_name, pipeline_version }
    ▼
FastAPI route (analytics_apps.py)
    │  IAnalyticsAppShim.start(params)
    ▼
ObjectDetectionAnalyticsAppShim
    │  resolves camera_id → RTSP URL via NxWitnessVmsShim
    │  POST /pipelines/{name}/{version}  →  DL Streamer Pipeline Server
    ▼
DL Streamer Pipeline Server (dls_vision)
    │  processes RTSP stream
    └─► inference results → MQTT broker  topic: /{vms_name}/dls_vision/{camera_id}
    ▼
MqttSubscriber (VAP background task)
    │  translate_dls_metadata() — DLS JSON → Nx analytics object format
    ▼
NxWitnessVmsShim.push_analytics_objects()
    │  POST /rest/v4/analytics/engines/{engine_id}/deviceAgents/{device_id}/metadata/object
    ▼
Nx Witness VMS
    └─► bounding boxes + labels overlaid on camera feed in Nx client

Key Components#

VMS Shims (vms_shim/)#

A class implementing the IVmsShim interface represents each VMS vendor:

Shim

Source

Camera Discovery

NxWitnessVmsShim

Nx Witness REST v4

Queries /rest/v4/devices

Camera IDs are vendor-prefixed strings (nx:abc123). The orchestrator uses the prefix to dispatch RTSP URL lookups and write-backs to the correct shim.

Analytics App Shims (analytics_app_shim/)#

A class implementing the IAnalyticsAppShim interface represents each AI analytics application:

Shim

App ID

Result Delivery

LiveCaptioningAnalyticsAppShim

live_captioning

SSE proxy to dashboard caption overlay

ObjectDetectionAnalyticsAppShim

dls_vision

MQTT → Nx Witness analytics objects

Adding a new Analytics App requires only a new shim class registered in plugin/core/factory.py. You do not need to change any routes.

FastAPI Backend (plugin/)#

The backend exposes a generic Analytics App API at /v1/analytics-apps/{app_id}/… for all integrations, plus camera management, event timeline, and health endpoints. Dependency injection via plugin/core/api/deps.py provides shim instances to all routes.

Orchestrator (plugin/core/pipeline/orchestrator.py)#

The orchestrator runs at startup to:

  • Construct and connect all VMS shims.

  • Register analytics manifests with Nx Witness.

  • Fetch Analytics App schemas (LVC OpenAPI, dls_vision pipeline list).

  • Start background tasks: camera sync loop, MQTT subscriber (for dls_vision).

Dynamic Schema (LVC)#

The LvcSchemaManager fetches the StartRunRequest JSON Schema from LVC’s /openapi.json at startup, resolves all $ref references, adds UI annotations, and builds a live Pydantic model. The dashboard renders analytics forms directly from this schema — the frontend does not need changes when LVC parameters change.

MQTT Subscriber (dls_vision)#

MqttSubscriber runs as an asyncio background task. It subscribes to +/dls_vision/+ on the MQTT broker and receives DL Streamer GStreamer Video Analytics (GVA) JSON metadata per frame. The translate_dls_metadata() function converts normalized bounding boxes and labels to Nx analytics object format, then NxWitnessVmsShim.push_analytics_objects() posts them to Nx Witness.

React Analytics Provider Dashboard (ui/)#

The nginx server serves the dashboard (React 19 with Vite and Tailwind CSS) and reverse-proxies:

  • /v1/* → FastAPI backend

  • /whep/* → MediaMTX (WebRTC video relay)

Key panels:

  • Camera Discovery: discover, enable, and disable cameras.

  • Analytics Engine: select an Analytics App, fill the dynamically rendered schema form, and start or stop runs.

  • Live Stream: WebRTC video player with caption overlay (LVC).

  • Analysis Results: timeline of metadata events.

Extensibility#

VAP supports extension:

  • Add a new VMS: implement IVmsShim in vms_shim/<vendor>/shim.py, register in factory.py.

  • Add a new Analytics App: implement IAnalyticsAppShim in analytics_app_shim/<name>/shim.py, register in factory.py. You do not need to change any routes.

Learn More#