Pipeline Performance#

This article describes how ViPPET collects and reports pipeline-level performance metrics: throughput (FPS), end-to-end latency and VLM generation metrics.

Throughput#

Pipelines run on DL Streamer Pipeline Server 2.0 (DLSPS), one server instance per stream. The server counts the frames of each instance at one terminal sink and reports them as frame_fps over the last frame_interval seconds in GET /pipelines/{id}/status. That sink is the one parsed last, so before starting an instance ViPPET moves one branch to the end of the launch string: the branch with gvafpscounter if exactly one branch has it and ends in a sink, otherwise the branch that ends in the stream’s main sink. In a VLM pipeline whose gvafpscounter sits after gvagenai, the FPS is therefore the rate of the VLM branch, not of the decode branch beside it. Dlsps2PipelineRunner (vippet/dlsps2_runner.py) polls the status of each instance every DLSPS2_POLL_INTERVAL_S seconds (default 1). After a warm-up of DLSPS2_WARMUP_S seconds (default 5, at most half the run), counted from the moment every stream has reported its first frame, it sums each stream’s frames and seconds over the new windows. A stream’s FPS is its frames over its seconds, and the slowest stream judges the run. While the run lasts, the runner pushes the mean FPS per stream to metrics-manager as the live FPS metric, tagged with the job_id.

Latency#

Latency measurement uses DL Streamer’s latency_tracer. It tracks how long each buffer takes to traverse the pipeline, from the source element to the sink element.

How latency_tracer is activated#

GStreamer tracers are set per process, so the tracer runs in the server for every pipeline the server runs, including tests that do not ask for latency metrics. It is on by default and set with environment variables of the vippet and dlstreamer-pipeline-server services in compose.yml:

Environment variable

Value

Purpose

DLSPS2_GST_TRACERS

latency_tracer(flags=pipeline,interval=1000,post-metrics=true)

Passed to the server as GST_TRACERS (default); empty turns it off

DLSPS2_OTEL

true (default)

Server exports its metrics over OpenTelemetry to metrics-manager

DLSPS2_OTEL_INTERVAL_MS

1000 (default)

Export interval of the server, in milliseconds

  • flags=pipeline — measures the total source-to-sink latency (as opposed to per-element latency).

  • interval=1000 — the tracer reports statistics accumulated during each 1 000 ms window.

  • post-metrics=true — the tracer posts every window as a dlstreamer/latency-window element message on the pipeline bus. The last, partial window is posted at end-of-stream.

A test with latency metrics enabled runs without them when DLSPS2_OTEL is not true or DLSPS2_GST_TRACERS does not run latency_tracer with post-metrics=true: a warning is logged and the job details say so. A tracer that predates post-metrics posts no messages, so no samples arrive, and the runner logs a warning after the run.

Latency samples#

The server merges the windows of each source-sink branch over one export period and exports them, tagged with the instance’s pipeline_id, source and sink, as the pipeline_latency_avg, _min, _max, _window and _frames gauges. Dlsps2PipelineRunner reads these series from metrics-manager every DLSPS2_METRICS_POLL_S seconds (default 1), only for its own instances, and derives one sample per export period:

Sample field

Value

Meaning

interval_ms

pipeline_latency_window

Total length of the windows in the export period

avg_ms

pipeline_latency_avg

Mean source-to-sink latency of a frame

min_ms

pipeline_latency_min

Minimum source-to-sink latency of a frame

max_ms

pipeline_latency_max

Maximum source-to-sink latency of a frame

latency_ms

pipeline_latency_window / pipeline_latency_frames

Deprecated. Frame period (1000 / fps), a throughput measure, not a latency

A frame’s source-to-sink latency runs from the moment the source element outputs the frame to the moment the frame is pushed into the sink, so the sink’s own work is excluded. latency_ms is the mean time between output frames in the period: it falls as throughput rises and says nothing about how long a frame spends in the pipeline. It is deprecated and kept only for API compatibility; use avg_ms for the mean latency and max_ms for the worst case.

After the instances stop, the runner keeps reading until their final export arrives, for at most DLSPS2_OTEL_INTERVAL_MS plus two seconds.

Data flow#

DL Streamer Pipeline Server 2.0 (one instance per stream)
   │  bus: dlstreamer/latency-window, dlstreamer/genai-request
   ▼
OpenTelemetry export (OTLP/JSON, every DLSPS2_OTEL_INTERVAL_MS)
   │
   ▼
metrics-manager  (series tagged with the server's pipeline_id)
   │
   ▼
Dlsps2PipelineRunner (reads its own instances every DLSPS2_METRICS_POLL_S)
   │
   ▼
pipeline_latency / vlm_metrics samples  ──▶  HTTP POST to metrics-manager:9090/api/v1/metrics
   │
   ▼
metrics-manager  ──▶  SSE stream at /metrics/stream
   │
   ▼
Browser (EventSource) → Redux store → Latency chart components
  1. DL Streamer Pipeline Server 2.0 — runs each stream as its own instance. With the setup above, it exports the latency windows and gvagenai requests of every instance to metrics-manager, tagged with the instance’s pipeline_id.

  2. Dlsps2PipelineRunner — reads only the series of its own instances, and republishes each export period as ViPPET’s own pipeline_latency and vlm_metrics samples with the job’s tags.

  3. metrics-manager — receives an HTTP POST with a pipeline_latency measurement (fields: avg_ms, min_ms, max_ms, latency_ms; tags: stream_id, job_id). It stores the sample and pushes it to connected SSE clients.

  4. UI (browser) — an EventSource connection to /metrics/stream dispatches incoming messages to the Redux store. The latency chart components render avg / min / max over time.

VLM generation metrics#

Pipelines with a gvagenai element that runs with metrics=true report the performance of generation requests. Dlsps2PipelineRunner asks the server for its gvagenai properties (GET /elements/gvagenai) and adds post-metrics=true to such an element only when the server’s gvagenai has the property, because an unknown property fails parsing. The element then posts a dlstreamer/genai-request message per request, and the server exports the genai_ttft, genai_tpot and genai_generate_duration histograms. The runner republishes them as a vlm_metrics sample with ttft_ms, tpot_ms and generate_duration_ms, tagged with the element name as stream_id. Each value is the mean over the requests of one export period.

Without the post-metrics property, a performance test falls back to the metrics block of the JSON answer each gvagenai writes, read through the event files of vippet/pipeline_events.py, with one warning per run. That block holds the means over every request since the element started, not per-period values. They are pushed as ttft_mean, tpot_mean and generate_duration_mean.