Customize the Application#

This comprehensive guide provides a detailed walkthrough of building a complete object tracking and loitering detection system. It utilizes a combination of technologies designed for ease of use: Deep Learning Streamer Pipeline Server (DL Streamer Pipeline Server) with native gvaanalytics zone and dwell-time analytics, and the data visualization Grafana platform. This approach caters to no-code/low-code users, enabling the creation of sophisticated analytics solutions with minimal programming.

Note

This page covers using the pipeline as shipped - zones, streams, models, and the published MQTT/Grafana data - with no config.json edits needed. To change the pipeline definition itself (add/remove a GStreamer element, change a hardcoded vs REST-overridable property), see Updating the Pipeline instead.

Overall System Architecture#

The system follows a modular architecture:

  • Video Input: Cameras or video streams provide the raw data.

  • Deep Learning Streamer Pipeline Server (DL Streamer Pipeline Server): Processes video streams locally using AI models to detect and track objects across the full frame, and evaluates each tracked object against configurable zone polygons to compute dwell time natively via the gvaanalytics element — no external data-processing service required.

  • Grafana: Visualizes the published MQTT data (zone presence, dwell time, and loitering status), providing real-time dashboards.

DL Streamer Pipeline Server#

Note

For detailed documentation on Deep Learning Streamer Pipeline Server (DL Streamer Pipeline Server), visit the DL Streamer Pipeline Server Documentation.

Pipeline Architecture

The DL Streamer Pipeline Server is a powerful tool designed to process video feeds directly on edge devices. It leverages GStreamer pipelines and OpenVINO™ optimized AI models to perform real-time object detection and tracking, minimizing latency and reducing bandwidth consumption.

Key DL Streamer Pipeline Server Components#

Logging and general configuration#

  • C_LOG_LEVEL: INFO - Sets the logging level for C components to “INFO,” which provides useful information about the system’s operations without being overly verbose.

  • PY_LOG_LEVEL: INFO - Similarly, sets the logging level for Python components.

Video processing pipelines#

The DL Streamer Pipeline Server utilizes GStreamer pipelines to define the flow of video data through various processing elements. This app defines a single family of pipelines that run detection, tracking, and zone/dwell-time analytics together (see below) — there is no separate detection-only pipeline stage.

Object tracking pipelines#

Pipelines object_tracking_cpu, object_tracking_gpu, and object_tracking_npu each run detection, tracking, and zone/dwell-time analytics in a single pipeline for a given device; a stream is launched against one of these by REST call (see sample_start.sh), which assigns it a per-instance MQTT topic and WebRTC peer-id (e.g. object_tracking_1).

  • Pipelines: object_tracking_cpu, object_tracking_gpu, object_tracking_npu

  • How They Work:

    • Detection Model: sample_start.sh always launches with the pedestrian-and-vehicle-detector-adas-0001 model; the Loitering Detection Console UI can launch any model placed under src/dlstreamer-pipeline-server/models/ instead, picked per stream at start time - no pipeline or config.json change needed either way. gvadetect runs with inference-region=0, so detection happens on the full frame, not just the configured zone polygons.

    • Tracking Element: Incorporates gvatrack with the setting tracking-type=zero-term to follow objects over time.

    • Zone & Dwell-Time Analytics: The gvaanalytics element evaluates each tracked object’s position against a per-stream polygon zone config file and computes dwell time; it is the only stage whose output is restricted to the configured zones — an object outside every zone is still detected and tracked, just not reported as present/dwelling in a zone. loitering_watermark (a reused DL Streamer sample element) renders the dwell-time/status text overlay on the video for every tracked object with dwell data, regardless of its detected class/label.

    • Additional Parameters:

      • Threshold: Set to 0.7 to balance sensitivity and accuracy.

      • Inference Interval: Controls how often the model analyzes frames.

  • Benefits: These pipelines help track moving objects (like people or vehicles) across frames, which is essential for loitering detection and other advanced analytics.

Configurable parameters#

Each pipeline is designed to be user-friendly and customizable:

  • Detection Properties: Configurable parameters let you adjust the model settings without writing code. Example: The “detection-properties” element defines how the object detection module should behave.

  • Detection Device: object_tracking_cpu/_gpu/_npu are separate pipeline definitions, one per device - the caller (sample_start.sh or the Console UI) selects which one to launch against by name, rather than one pipeline taking a device parameter at runtime. See Updating the Pipeline for the full REST parameter reference.

Messaging Interface (MQTT)#

To share the results of the video analysis with other parts of your system, the DL Streamer Pipeline Server uses a messaging interface based on Message Queuing Telemetry Transport (MQTT):

  • Publisher Configuration:

    • Name: default

    • Type: mqtt

    • Endpoint: tcp://0.0.0.0:1883

    • Topics: Each launched stream publishes its raw per-frame metadata under object_tracking/<N> (set per-instance via destination.metadata.topic in the REST launch payload, not hardcoded; see sample_start.sh). The mqtt-table-flattener sidecar republishes a reshaped, threshold-filtered summary to object_tracking/<N>/loiterstatus for the Grafana table.

    • Allowed Clients: * (* denotes ‘all’, ensuring that any subscribed system can receive the data)

DL Streamer Pipeline Server workflow#

  1. Capture and Decode: Live video is captured and decoded using GStreamer.

  2. Detection and Tracking: The object detection pipelines analyze each frame to identify objects. The tracking pipelines then follow these objects over time, ensuring that moving objects are continuously monitored.

  3. Data Enrichment and Publishing: Detected objects are enriched with metadata (such as bounding boxes, timestamps, and object types). The processed data is published via MQTT, making it available for dashboards.

Zone & Dwell-Time Analytics (gvaanalytics)#

Note

For detailed documentation on the gvaanalytics element, visit the DL Streamer documentation.

Zone presence and dwell-time computation run natively inside the DL Streamer pipeline via the gvaanalytics element — no external low-code tool or custom service is required. This keeps the whole pipeline (detection, tracking, zone/dwell analytics, and the on-screen watermark) in one process, and the resulting data already flows to MQTT through the pipeline’s existing gvametaconvert + destination.metadata publish path.

gvadetect (inference-region=0) and gvatrack always run on the full video frame — there is no ROI-cropping element in this pipeline, so every object in frame is detected and tracked regardless of the zone polygons. gvaanalytics is the stage that narrows scope: it only reports zone presence and dwell time for the polygon region(s) defined in the stream’s zone-config file; objects outside those polygons are detected/tracked but produce no zone/dwell metadata.

Zone configuration files#

Each stream has its own zone-config JSON file under src/dlstreamer-pipeline-server/zones/, for example VIRAT_S_000101.json:

{
  "zones": [
    {
      "id": "region1",
      "type": "polygon",
      "points": [
        { "x": 208, "y": 390 },
        { "x": 2, "y": 489 },
        { "x": 2, "y": 296 },
        { "x": 197, "y": 237 }
      ],
      "track-dwell-time": true,
      "object-retention": 1.0,
      "color": { "r": 0, "g": 255, "b": 0 },
      "thickness": 2
    }
  ]
}
  • id: Zone identifier, reported as zone_id in the published metadata.

  • type: "polygon" (arbitrary number of points) or "circle".

  • track-dwell-time: Enables dwell-time computation for this zone.

  • object-retention: Grace period (seconds) to keep zone state after an object leaves.

  • color/thickness: Used when rendering the zone outline on the video overlay.

A zone file can contain any number of zones — adding a second or third zone to a stream is purely a matter of adding another entry to the zones array in that stream’s JSON file.

Adding or editing a zone (no code changes)#

  1. Edit (or add zones to) the stream’s JSON file under src/dlstreamer-pipeline-server/zones/.

  2. Restart that stream’s pipeline instance so it picks up the change (gvaanalytics only reads its config file once, when the pipeline starts — editing the file does not hot-reload a running pipeline): bash ./sample_stop.sh ./sample_start.sh No config.json, pipeline string, or Grafana dashboard changes are needed.

Adding a new stream#

  1. Add a video file under src/dlstreamer-pipeline-server/videos/ and a matching zone-config JSON file under src/dlstreamer-pipeline-server/zones/.

  2. Launch it with one more REST call (see sample_start.sh for the pattern), passing "analytics-properties": {"config": "/home/pipeline-server/zones/<your-zone-file>.json"} in the parameters object, alongside a unique destination.metadata.topic / destination.frame.peer-id for the new stream.

  3. Add the new stream name to the stream dashboard variable in the Grafana dashboard so its video panel is rendered (the MQTT status table requires no change — it already subscribes to a wildcard topic covering all streams).

Published metadata#

gvametaconvert (already part of the pipeline) natively serializes each tracked object’s zone membership and dwell time into the existing MQTT payload on object_tracking/<N>, per object:

{
  "objects": [
    {
      "id": 7,
      "x": 120,
      "y": 240,
      "w": 40,
      "h": 90,
      "roi_type": "pedestrian",
      "zone_violations": ["region1"],
      "dwell_times": [
        {
          "zone_id": "region1",
          "dwell_time_sec": 6.2,
          "first_seen_timestamp_sec": 118.4
        }
      ]
    }
  ],
  "resolution": { "width": 1280, "height": 720 },
  "timestamp": 123456789
}

No custom metadata-publishing code is needed — this is the same MQTT topic and publish mechanism the application already uses for object detection/tracking data. object_tracking/<N> is never modified by this application and remains available as-is for any external integration that wants the raw, full per-frame payload.

Grafana table data (mqtt-table-flattener)#

Grafana’s MQTT data source maps one MQTT message to one table row, and cannot expand a nested JSON array (like objects above) into multiple rows on its own. A small sidecar service, mqtt-table-flattener (src/mqtt-table-flattener/flatten_and_filter.py), bridges this gap:

  • Subscribes read-only to object_tracking/+ (never alters it).

  • Performs no zone/dwell-time computation itself — all of that is still done exclusively by gvaanalytics; the flattener only reshapes already-computed values for display.

  • Once a second, republishes one small flat message per currently-dwelling object whose dwell time has reached LOITERING_THRESHOLD_SEC to <source_topic>/loiterstatus, e.g. object_tracking/1/loiterstatus → {"Stream": "1", "Track ID": 7, "Type": "pedestrian", "Zone": "region1", "Dwell Time (s)": 6.2}. This mirrors the same threshold loitering_watermark.py uses to color overlay text red, so only objects shown red in the video overlay appear as table rows.

  • Stops republishing a stream once it has been silent for a few seconds (STALE_AFTER_SEC, default 3s), so the table correctly empties out after a pipeline stops instead of repeating its last-known state forever.

  • Runs as a second container from the same dlstreamer-pipeline-server image (already pulled for the main pipeline service, and already has paho-mqtt) — no extra image pull and no pip install at runtime, so it works in air-gapped deployments too.

  • The topic suffix and threshold are not hardcoded: they come from SOURCE_TOPIC_PREFIX, DEST_TOPIC_SUFFIX, and LOITERING_THRESHOLD_SEC — static entries in the root .env file. These .env vars are loitering-detection-specific and unused by smart-parking/smart-intersection, so they have no effect on those apps.

The Grafana dashboard’s status table subscribes to object_tracking/+/loiterstatus and needs no per-stream changes when adding a new stream, since it already covers the whole topic namespace.

Video overlay#

The reused loitering_watermark element reads the same dwell-time metadata and draws a per-object dashboard line (<zone_id>: <type>-<id> : <dwell_time>s) for every tracked object that has zone/dwell data, regardless of its detected class name (so it works the same whether the launched model reports pedestrian, person, or anything else) - turning red once dwell time crosses the configured loitering-threshold (default 5.0 seconds).

Tune or disable the dwell-time overlay#

The on-screen dwell-time dashboard text (drawn by loitering_watermark) is controlled by two REST-overridable properties, set via loitering-watermark-properties in the launch payload used by sample_start.sh — no code or pipeline changes needed:

  • loitering-threshold: dwell time (seconds) after which the overlay turns red. Default 5.0. To change it, edit the "loitering-threshold" value in the payload in sample_start.sh (or pass a different value in your own curl call using the same payload shape), then restart the pipelines so the new value takes effect:

    ./sample_stop.sh
    ./sample_start.sh
    

    Note

    Pipeline parameters are only read when a pipeline starts, so editing the payload value alone does not affect an already-running pipeline — it must be restarted.

  • quiet-mode: set to "true" to suppress the on-screen text entirely (e.g., to rely on the Grafana table only); "false" to re-enable it. Example REST payload fragment (see sample_start.sh for the full launch payload):

    {
      "parameters": {
        "loitering-watermark-properties": {
          "loitering-threshold": 7.0,
          "quiet-mode": "false"
        }
      }
    }
    

    Important

    loitering-threshold must be a JSON number (e.g., 7.0), not a quoted string ("7.0").

    This only affects the on-screen text; zone outlines/bounding boxes and the Grafana table are unaffected.

    For Console-launched pipelines, select Show dwell-time overlay on video before starting them; stop and restart an existing pipeline to change its overlay setting. The Console uses its own LOITER_THRESHOLD_S setting in console-addon/.env for both its status table and the overlay threshold of pipelines it launches. Stop Console pipelines before recreating the console service, which clears its session state. After changing the setting, run docker compose up -d --force-recreate console and launch new pipelines. Changing a REST overlay property alone does not change the Console’s status threshold. See Use the Console UI.

Grafana visualization#

Note

For detailed Grafana documentation, visit the Official Grafana Documentation.

Grafana Dashboard

Grafana is a powerful, open-source visualization tool that helps you create dynamic dashboards for monitoring real-time data. With Grafana, you can easily visualize the outputs from your DL Streamer Pipeline Server without deep coding skills. Here is how you can leverage Grafana in your analytics workflow.

Key Grafana Components#

Data Sources#

  • MQTT Integration: Directly query data from the DL Streamer Pipeline Server via MQTT datasource.

Dashboards and Panels#

  • Dashboards: A collection of panels arranged to provide an overview of your system’s performance.

  • Panels: Individual visualizations (graphs, tables, single-stat panels) that display specific data points.

Setting up Grafana#

Install and launch Grafana#

  • Installation: Install Grafana on your local machine or server by following the official Grafana installation guide.

  • Access: Once installed, access Grafana via your web browser at the designated URL.

Add Your data source#

  • Step-by-Step (Example with MQTT):

    1. Go to Configuration > Data Sources.

    2. Click Add data source and select MQTT.

    3. Enter the connection details (host, port, topic, credentials).

    4. Save and test the connection to ensure it works.

Create your dashboard#

  • Dashboard Creation:

    1. Click the + icon and select Dashboard.

    2. Add panels by clicking Add new panel.

    3. Configure each panel with queries to fetch data from your connected MQTT data source.

    4. Use drag-and-drop controls to arrange panels for the best view of your metrics.

Grafana use cases for object tracking and Loitering Detection#

  • Real-Time Object Detection Visualization: Display a graph that shows the number of objects detected per minute to monitor activity levels.

  • Loitering Event Monitoring: Create a panel that highlights loitering events.

  • Historical Trend Analysis: Use Grafana’s time-series graphs to analyze trends over days or weeks, helping you identify peak activity times or recurring patterns.

End-to-End integration#

The system operates as follows:

  1. Video Input: A camera captures video and sends the stream to the DL Streamer Pipeline Server.

  2. DL Streamer Pipeline Server Processing: The DL Streamer Pipeline Server processes the video, detects and tracks objects using its AI models, and evaluates each tracked object against configured zone polygons via gvaanalytics to compute zone presence and dwell time. It publishes metadata about the detected objects (ID, bounding box coordinates, object type, timestamps, zone presence, and dwell time) to MQTT.

  3. MQTT Bridging (DL Streamer Pipeline Server Configuration): The DL Streamer Pipeline Server is configured to relay the MQTT messages to an MQTT broker. This broker acts as a central hub for the data.

  4. Grafana Visualization: Grafana directly consumes MQTT topics through its MQTT datasource to create dashboards showing real-time object counts and loitering events.

Data at each step:

  • DL Streamer Pipeline Server Output (MQTT): JSON payload containing an array of detected objects. Each object has properties like id, roi_type, confidence, bounding box coordinates, zone_violations, and dwell_times (zone ID, dwell time, and first-seen timestamp).

Conclusion#

This section has demonstrated a complete object tracking and loitering detection system using DL Streamer Pipeline Server (with native gvaanalytics zone/dwell-time analytics) and Grafana. The system provides a balance of edge processing and powerful visualization, with zone configuration handled entirely through JSON files — no custom data-processing service, flow duplication, or per-zone code is required. This allows for quick deployment, easy customization for specific use cases, and scalability to handle multiple cameras and zones.