gvastreammux#
Muxes multiple streams into a single pipeline so that one downstream consumer can serve N input
sources, replacing N independent pipelines and significantly reducing GPU memory usage. For
all-video inputs this means a shared gvadetect; the element also supports heterogeneous
inputs (e.g. video + lidar). Each output buffer is tagged with GstAnalyticsBatchMeta so that
gvastreamdemux can later route it back to the correct source.
Overview#
gvastreammux collects frames from multiple sink pads into per-pad queues, assembles
PTS-aligned batches, and emits each batch downstream. The single src pad adapts its output
to the mix of sink-pad media types — see Output Modes. Every output buffer
carries a GstAnalyticsBatchMeta whose streams[].index identifies the originating pad(s) and
n_streams records the batch size.
Key design points:
PTS-anchor batch assembly — for each batch the element picks the earliest PTS across all non-EOS pads as the anchor, then waits up to
max-wait-timefor the other pads to deliver buffers whose PTS lies withinpts-toleranceof that anchor. Pads that miss the window do not block the batch indefinitely; they simply do not contribute to that cycle.Back-pressure on upstream — each pad has its own bounded queue (
max-queue-size). When a queue is full, the upstream chain function blocks until the output loop drains it. This prevents unbounded memory growth without dropping frames.Late-frame dropping — if a buffer’s PTS is more than
pts-tolerancebehind the most recent pushed batch, it is dropped (the batch already moved past that time and there is no way to insert it).Per-pad PTS normalization (
sync-mode) — when sources have unrelated PTS timelines (e.g. files starting at different timestamps, multiple cameras with independent clocks), the element can normalize PTS before scheduling. See the Sync Mode section.Sparse pad indices — names like
mux.sink_5work even ifsink_0..sink_4were never created. Indices must be in[0, 256).
Important: each source contributes one buffer per batch, so a downstream
gvadetect inference-interval=N(with N > 1) skips every Nth buffer in arrival order, which means certain source ids will always be skipped. Useinference-interval=1(the default) when the goal is uniform coverage across all input streams.
How It Works#
The pipeline requests sink pads (
sink_0,sink_1, …) — one per input source. Pad index is parsed from the requested name; sparse indices are allowed up to 255. Request a pad without a name to be given the lowest free index.Upstream chain functions push buffers into per-pad queues. If a pad’s queue is full, the chain function blocks until the output task drains it (back-pressure to upstream).
The output task on the src pad assembles batches in three phases:
Phase 1 (anchor selection) — discard any queue head older than the batch already pushed, then pick the earliest valid PTS across all non-EOS pad queue heads as
batch_anchor_pts. Discarding first keeps the anchor monotonic: a buffer older than the last batch has missed it, and taking it as the next anchor would move batch time (and, incontainermode, the output buffer’s PTS) backwards.Phase 2 (wait for contributors) — wait up to
max-wait-timefor every active non-EOS pad to have a head buffer with|pts - anchor| ≤ pts-tolerance. Exit early once they all contribute. See Slow and Failed Sources.Phase 3 (collect & push) — pop one matching buffer per pad, attach
GstAnalyticsBatchMeta, push downstream in pad-index order.
EOS on a sink pad is honored only once that pad’s queue is drained: buffers already enqueued before EOS are still batched (so the last frames of a stream, or a single-frame source, are not lost). A pad with no remaining buffers and EOS set is excluded from future batches. When all pads are EOS and all queues are drained, the src pad emits EOS and the task pauses.
The muxer presents a single stream assembled from several, so it emits exactly one
stream-startwith its own stream id. Each sink pad’s upstreamstream-startis consumed rather than forwarded.Per-pad
FLUSH_START/FLUSH_STOPevents are coalesced — the downstream flush, task pause, queue reset and task restart each happen exactly once per flush cycle no matter how many sink pads receive the events. A flush removes the sticky segment from thesrcpad, so the muxer sends a replacement immediately afterwards.
Slow and Failed Sources#
A batch waits only for the sources that are behaving. Each sink pad is in one of three states, and a pad leaves the first one as soon as anything arrives on it — recovery is judged on delivery, not on whether what was delivered could be used.
State |
Entered when |
Effect |
|---|---|---|
|
buffers are arriving and being batched |
batches wait for this pad |
|
nothing has arrived for five batches |
batches no longer wait for it |
|
its queue head has been discarded as stale for three batches running |
batches no longer wait for it |
Neither unhealthy state excludes a pad from batching — its buffers still go out whenever they match the anchor. The pad simply stops holding everyone else up, so a dead feed costs its own share of the throughput and nothing more. With four sources at 30 fps and one of them cut off, output goes from 120 to 90 buffers per second rather than collapsing to 74.
stalled and late are separate because they need different responses: nothing is arriving
in the first, while in the second the source is alive and working through a backlog that
Phase 1 discards until it is current again.
The thresholds are counted in batches, not milliseconds, so they scale with max-wait-time
— already your statement of how long a straggler may hold a batch open.
Every change of state is posted on the bus as a GST_MESSAGE_ELEMENT, once per transition:
gva-streammux-stream-state, pad-index=(uint)3, state=(string)stalled,
active-pads=(uint)3, total-pads=(uint)4
state is active, stalled or late. Watch for it to find out which source went bad
without turning on debug logging.
Adding and Removing Sources at Runtime#
Sink pads can be requested and released while the pipeline is running.
Releasing a pad that has already reached EOS does not end the stream: the muxer only emits EOS once every pad that is still attached has finished. Releasing finished sources one at a time, as a long-running pipeline recycles them, therefore keeps the live ones running.
A pad may be released while its upstream is still pushing. Its chain function is told to stop first, so a source that is still delivering is not left waiting for queue space that will never be freed.
Requesting an index that is already in use is rejected (
request_new_padreturnsNULL) rather than replacing the existing pad. Request pads without a name to get the first free index.If every pad reaches EOS before any of them reported caps, the source caps can never be negotiated. The muxer ends the stream itself, sending stream-start and a segment (and, in
containermode, the batch caps) ahead of the EOS.A source that goes quiet, or comes back delivering buffers that are already too old, stops holding up the batches of the sources that are still healthy. See Slow and Failed Sources.
Output Modes#
The output mode is selected explicitly by the output-mode property (default passthrough);
there is no auto-detection from the input media types. The element negotiates its src caps
once, after every live sink pad has reported its caps, and the output loop does not push
anything until then.
Mode ( |
Output |
|---|---|
|
Each source buffer is pushed as-is, tagged with a single-stream |
|
Each assembled batch is packed into one container buffer with caps |
Important (
containermode): a container buffer is not raw video, so no video element (gvadetect,gvawatermark, …) may follow the mux directly. Insertgvastreamdemuxto recover the per-source streams (e.g. video →gvadetect, lidar →g3dinference) first.
Caps changes mid-stream (
passthroughmode): thesrccaps are fixed at negotiation and cannot be changed afterwards, because buffers already queued were accepted under the old caps. If a sink pad reports different caps later — an RTSP source reconnecting at another resolution, for instance — the element posts an error instead of silently forwarding mismatched buffers. Usecontainermode for sources whose caps can change.
Choosing the mode: use
passthroughwhen every source produces the same caps (the common all-video case) and you want plain buffers downstream. Usecontainerwhenever the sources are heterogeneous (mixed media types or differing caps) — apassthroughmux with mismatched sink caps will error out at caps negotiation.
passthrough mode keeps the historical one-buffer-per-source-per-batch shape; container mode is
one-buffer-per-batch carrying all sources.
Properties#
Property |
Type |
Default |
Description |
|---|---|---|---|
|
Double |
|
Output rate cap on the assembled batches (0 = unlimited). Only set for local file sources; setting on RTSP/live sources can stall the pipeline. |
|
UInt64 (ns) |
|
Max |
|
UInt64 (ns) |
|
Max time the output task waits for late pads after the anchor is set. After timeout the partial batch is pushed. |
|
UInt |
|
Maximum buffers per pad queue. When reached, upstream blocks (back-pressure). |
|
Enum |
|
How to normalize PTS across pads. See Sync Mode. |
|
Enum |
|
|
Sync Mode#
Multi-source pipelines often deliver buffers with PTS values that cannot be compared directly
(different file starts, multiple cameras with independent clocks, etc.). Without normalization
the anchor + tolerance scheduler either deadlocks (a far-future pad never matches the window
and back-pressures upstream) or oscillates (anchor jumps each round, batches end up
single-stream). sync-mode selects the normalization policy.
Mode |
What it does |
Typical use case |
|---|---|---|
|
Use buffer PTS as-is. Default; preserves legacy behavior. |
Single source, or upstream already aligned (e.g. |
|
Subtract each pad’s first observed PTS so every pad starts at 0. Survives a source restarting its timeline (see below). |
Multiple file sources whose PTS bases differ; RTSP sources that reconnect. |
|
Subtract each pad’s |
Standard GStreamer pipelines, including seek/trick mode. |
|
Overwrite PTS with |
Single-machine multi-live source where source PTS are unreliable. |
|
Replace PTS with |
Cross-device cameras synchronized via NTP/PTP, paired with |
Per-pad normalization state (first PTS, segment start) is reset on FLUSH_STOP and on
PAUSED → READY, so seek and pipeline restart establish fresh baselines.
first-pts additionally handles a source whose timeline restarts mid-run, which is what an
RTSP feed does when it reconnects: PTS begins from zero again, with DISCONT set. The pad’s
baseline is re-derived so its normalized time continues from where it left off rather than
restarting at zero — a pad that went back to zero while the others were seconds ahead would
have every buffer discarded as a late frame and would never recover. A restart is recognised
only when a buffer is flagged DISCONT and its PTS jumps by more than a second; either
signal alone (a dropped frame, a sparse source) leaves the baseline untouched.
The segment on the src pad follows from the mode. first-pts, segment and pipeline
rewrite every PTS onto a zero-based timeline, so the muxer sends a zero-based segment to match.
none passes PTS through untouched, so it forwards the upstream segment instead — otherwise a
downstream sink would read a buffer seeked to T as having a running time of T and wait that
long before rendering it. A flush starts a new run, so a seek adopts the new segment.
Buffer Metadata#
Each output buffer carries a GstAnalyticsBatchMeta. The way it is populated depends on the
output mode:
PASSTHROUGH (video-only): one buffer per source, single-stream meta.
Field |
Type |
Description |
|---|---|---|
|
guint |
Source pad index this buffer originated from. |
|
gsize |
Always |
Up to and including the 2026.2 release this reported the size of the batch the buffer belonged to while still filling in only
streams[0], so consumers that walkstreams[0..n_streams)—gvametaconvert,g3drender,g3dobjectfuser— sawbatch_size - 1zeroed entries on every buffer. How many sources were in the batch is a different fact and is not carried here.
CONTAINER (heterogeneous): one container buffer per batch, n_streams entries.
Field |
Type |
Description |
|---|---|---|
|
gsize |
Number of sources packed in this batch. |
|
guint |
Source pad index of the i-th packed stream. |
|
GstBuffer |
The i-th source’s actual buffer. |
|
GstEvent |
The i-th source’s original caps (a |
gvastreamdemux reads this meta to route each source’s buffer to the matching src_* pad.
Pipeline Examples#
Local files with max-fps#
gst-launch-1.0 \
gvastreammux name=mux max-fps=30 \
! queue \
! gvadetect model=model.xml device=GPU \
pre-process-backend=va-surface-sharing \
! queue ! gvafpscounter ! fakesink \
filesrc location=video0.h265 ! h265parse ! vah265dec ! mux.sink_0 \
filesrc location=video1.h265 ! h265parse ! vah265dec ! mux.sink_1
Multiple files with different PTS bases (sync-mode=first-pts)#
gst-launch-1.0 \
gvastreammux name=mux sync-mode=first-pts max-fps=30 \
! queue \
! gvadetect model=model.xml device=GPU \
! queue ! gvafpscounter ! fakesink \
filesrc location=clip-recorded-at-10am.mp4 ! qtdemux ! h265parse ! vah265dec ! mux.sink_0 \
filesrc location=clip-recorded-at-2pm.mp4 ! qtdemux ! h265parse ! vah265dec ! mux.sink_1
RTSP sources, no max-fps#
gst-launch-1.0 \
gvastreammux name=mux \
! queue \
! gvadetect model=model.xml device=GPU \
pre-process-backend=va-surface-sharing \
! queue ! gvafpscounter ! fakesink \
rtspsrc location=rtsp://host:8554/stream latency=200 \
! rtph265depay ! h265parse ! vah265dec ! mux.sink_0 \
rtspsrc location=rtsp://host:8555/stream latency=200 \
! rtph265depay ! h265parse ! vah265dec ! mux.sink_1
Multi-camera NTP/PTP synchronization (sync-mode=ntp)#
When sources are real IP cameras synchronized via NTP (and the SDP advertises an RFC7273
clock), set rtspsrc add-reference-timestamp-meta=true so each buffer carries a
GstReferenceTimestampMeta with absolute time, then let the mux use that as PTS:
gst-launch-1.0 \
gvastreammux name=mux sync-mode=ntp pts-tolerance=33000000 \
! queue ! gvadetect model=model.xml device=GPU \
! queue ! gvafpscounter ! fakesink \
rtspsrc location=rtsp://cam1 ntp-sync=true add-reference-timestamp-meta=true \
! rtph265depay ! h265parse ! avdec_h265 ! mux.sink_0 \
rtspsrc location=rtsp://cam2 ntp-sync=true add-reference-timestamp-meta=true \
! rtph265depay ! h265parse ! avdec_h265 ! mux.sink_1
pts-tolerance is widened above the typical inter-frame interval to accommodate per-camera
network jitter; tune it for your environment.
Pair with gvastreamdemux for per-source output#
gst-launch-1.0 \
gvastreammux name=mux max-fps=30 \
! queue \
! gvadetect model=model.xml device=GPU \
! gvastreamdemux name=demux \
demux.src_0 ! queue ! gvafpscounter ! fakesink \
demux.src_1 ! queue ! gvafpscounter ! fakesink \
filesrc location=video0.h265 ! h265parse ! vah265dec ! mux.sink_0 \
filesrc location=video1.h265 ! h265parse ! vah265dec ! mux.sink_1
Heterogeneous sources: video + lidar (CONTAINER mode)#
Heterogeneous sources (here lidar via g3dlidarparse alongside video) require
CONTAINER mode, selected with output-mode=container. The container batch must be
unpacked by gvastreamdemux before any per-stream processing — no gvadetect may sit between the
mux and demux. In this example each demuxed branch goes straight to fakesink (src_0/src_1 =
video, src_2 = lidar):
gst-launch-1.0 -e \
gvastreammux name=mux output-mode=container sync-mode=first-pts \
! queue \
! gvastreamdemux name=demux \
demux.src_0 ! queue ! gvafpscounter ! fakesink \
demux.src_1 ! queue ! gvafpscounter ! fakesink \
demux.src_2 ! queue ! fakesink \
filesrc location=video0.h265 ! h265parse ! vah265dec ! mux.sink_0 \
filesrc location=video1.h265 ! h265parse ! vah265dec ! mux.sink_1 \
multifilesrc location=velodyne/%06d.bin start-index=0 caps=application/octet-stream \
! g3dlidarparse frame-rate=10 ! mux.sink_2
Lidar typically runs at a lower rate (e.g. 10 fps) than video (e.g. 30 fps). With aligned PTS, batches that have a coincident lidar frame carry it (
n_streamsincludes the lidar pad); video-only intervals produce video-only batches. Ensure the lidar branch produces valid PTS (g3dlidarparse frame-rate=...) and pick async-modeso the two timelines are comparable.
Tuning Notes#
pts-toleranceshould comfortably accommodate the largest expected inter-source PTS drift after normalization. For 30 fps sources a value of 16–33 ms is a reasonable starting point.max-wait-timebounds the worst-case batch latency when one pad stops delivering. Set it somewhat above one frame interval (e.g. 40–50 ms for 30 fps) so transient stalls do not immediately break the batch.max-queue-sizecontrols the back-pressure depth. Larger values absorb more upstream bursts at the cost of memory; 2–4 is usually enough.
Element Details (gst-inspect-1.0)#
Factory Details:
Long-name GVA Stream Muxer
Klass Muxer
Description Muxes multiple streams with PTS-based synchronization. The
'output-mode' property selects between PASSTHROUGH (each source
buffer pushed as-is; all sink pads must share identical caps) and
CONTAINER (each batch packed into one multistream/x-analytics-batch
buffer, e.g. video + lidar).
Author Intel Corporation
Pad Templates:
SINK template: 'sink_%u' (On request, video/x-raw, video/x-raw(memory:VAMemory) NV12,
video/x-raw(memory:DMABuf) DMA_DRM, application/x-lidar,
application/x-radar-processed; on Windows
video/x-raw(memory:D3D11Memory) and
video/x-raw(memory:D3D12Memory) NV12/BGRA instead of
VAMemory and DMABuf)
SRC template: 'src' (Always, the sink caps above (passthrough) OR
multistream/x-analytics-batch(meta:GstAnalyticsBatchMeta) (container))
Element Properties:
max-fps Double, range 0-Inf, default 0
pts-tolerance UInt64 ns, default 20000000 (20 ms)
max-wait-time UInt64 ns, default 40000000 (40 ms)
max-queue-size UInt, range 1-..., default 2
sync-mode Enum (none, first-pts, segment, pipeline, ntp), default none
output-mode Enum (passthrough, container), default passthrough
Run gst-inspect-1.0 gvastreammux against your installation for the authoritative output.