Structured Metrics Messages#
Some DL Streamer elements can post their measurements on the pipeline bus as GStreamer
element messages. An application then reads typed values from a GstStructure instead of
parsing log lines or JSON metadata. The values are the element’s own measurements, so the
application does not need to recompute them.
Posting is opt-in. Each producer has a post-metrics setting, which is false by default.
When it is not set, nothing is posted and the element’s other output (logs, tracer records,
JSON metadata) is unchanged.
Messages are posted once per request or once per aggregation window, never per frame or per token.
Message format#
The message type is
GST_MESSAGE_ELEMENT, and its source is the element that measured the values.The structure name is
dlstreamer/<kind>, and identifies which fields to expect.Every structure has a
versionfield (uint). The version is increased when a field is removed or changes its meaning or type. Fields may be added without a version change.Units are part of the field name, for example
_msfor milliseconds. Counts are integers.Messages carry no timestamp. Use the time the message is received.
A consumer should process the versions it knows, ignore fields it does not know, and skip (and report) a message whose version it does not know.
Messages#
Structure name |
Posted by |
Enable with |
Posted |
|---|---|---|---|
|
|
Once per interval window per source-to-sink branch |
|
|
|
Once per generation request |
dlstreamer/latency-window#
One latency_tracer interval window of a source-to-sink branch. It is posted by the branch’s
sink element when the tracer logs the matching latency_tracer_pipeline_interval record,
and only when the tracer flags include pipeline. The window that is still open when EOS
reaches the sink is posted as well, so the windows of a run cover all its frames.
The frame latency is the time from the source element pushing a frame to the frame being pushed into the sink, so the sink’s own work is not included. The message describes the frame latencies of the window as a count, a sum, the extremes, a sum of squares and a histogram.
Field |
Type |
Description |
|---|---|---|
|
uint |
Schema version, |
|
string |
Name of the pipeline |
|
string |
Name of the branch’s source element |
|
string |
Name of the branch’s sink element |
|
double |
Actual length of the window |
|
uint |
Number of frames that reached the sink in the window |
|
double |
Sum of the frame latencies |
|
double |
Minimum frame latency |
|
double |
Maximum frame latency |
|
double |
Sum of the squared frame latencies, in ms² |
|
array of double |
Increasing upper bounds of the histogram buckets |
|
array of uint |
Frames per bucket, one more entry than |
bucket_counts[0] counts the frames with a latency up to bucket_bounds_ms[0],
bucket_counts[i] those above bucket_bounds_ms[i-1] and up to bucket_bounds_ms[i], and
the last entry those above the last bound, as in OpenTelemetry explicit-bucket histograms.
The bounds currently run from 1 ms to 100 s, each about 25 % above the previous one. Read
them from the message instead of assuming them.
What can be derived from a window:
Value |
Derivation |
|---|---|
Mean frame latency |
|
Standard deviation of the frame latency |
|
Frame latency percentiles, e.g. p95 |
From the histogram: find the bucket that holds rank |
Frame rate |
|
Frame period |
|
Windows of several intervals or branches can be combined exactly: add up count, sum_ms,
sum_sq_ms, window_ms and the bucket_counts (when the bounds are the same), and take the
minimum of min_ms and the maximum of max_ms. Averaging the per-window means or
percentiles instead would weight windows equally regardless of how many frames they hold.
dlstreamer/genai-request#
Performance of one gvagenai generation request, as measured by OpenVINO™ GenAI. The values
describe that request only. The metrics block of the element’s JSON metadata, in contrast,
holds averages over every request since the element started.
Only the openvino-genai backend measures these values. With the openai-http backend no
message is posted, and the element logs a warning once.
Field |
Type |
Description |
|---|---|---|
|
uint |
Schema version, |
|
uint |
Number of frames the request was generated from |
|
double |
Time to first token |
|
double |
Time per output token, averaged over this request’s decode steps |
|
double |
Duration of the whole |
|
double |
Time spent in model inference |
|
double |
Time spent preparing the image and text embeddings |
|
uint64 |
Number of input tokens, including image tokens |
|
uint64 |
Number of generated tokens |
Reading messages in Python#
import gi
gi.require_version("Gst", "1.0")
from gi.repository import GLib, Gst
# Tracers are created by Gst.init(), so set GST_TRACERS before it runs, e.g.
# GST_TRACERS="latency_tracer(flags=pipeline,interval=1000,post-metrics=true)"
Gst.init(None)
pipeline = Gst.parse_launch("videotestsrc num-buffers=300 ! video/x-raw,framerate=30/1 ! fakesink sync=true")
loop = GLib.MainLoop()
def on_message(bus, message):
if message.type == Gst.MessageType.ELEMENT:
s = message.get_structure()
if s.get_name() == "dlstreamer/latency-window":
if s.get_value("version") != 1:
print(f"skipping unknown {s.get_name()} version {s.get_value('version')}")
else:
count = s.get_value("count")
print(f"{s.get_value('sink')}: avg latency {s.get_value('sum_ms') / count:.2f} ms, "
f"{1000 * count / s.get_value('window_ms'):.1f} fps")
elif s.get_name() == "dlstreamer/genai-request": # from a gvagenai element with post-metrics=true
if s.get_value("version") != 1:
print(f"skipping unknown {s.get_name()} version {s.get_value('version')}")
else:
print(f"TTFT {s.get_value('ttft_ms'):.1f} ms, TPOT {s.get_value('tpot_ms'):.1f} ms")
elif message.type in (Gst.MessageType.EOS, Gst.MessageType.ERROR):
loop.quit()
bus = pipeline.get_bus()
bus.add_signal_watch()
bus.connect("message", on_message)
pipeline.set_state(Gst.State.PLAYING)
loop.run()
pipeline.set_state(Gst.State.NULL)
From the command line, gst-launch-1.0 -m prints every posted message.
The latency_metrics
sample shows a complete application that reads dlstreamer/latency-window messages and
merges them into totals per branch.