# gvastreammux Muxes multiple streams into a single pipeline so that one downstream consumer can serve N input sources, replacing N independent pipelines and significantly reducing GPU memory usage. For all-video inputs this means a shared `gvadetect`; the element also supports **heterogeneous** inputs (e.g. video + lidar). Each output buffer is tagged with `GstAnalyticsBatchMeta` so that `gvastreamdemux` can later route it back to the correct source. ## Overview `gvastreammux` collects frames from multiple sink pads into per-pad queues, assembles **PTS-aligned batches**, and emits each batch downstream. The single `src` pad adapts its output to the mix of sink-pad media types — see [Output Modes](#output-modes). Every output buffer carries a `GstAnalyticsBatchMeta` whose `streams[].index` identifies the originating pad(s) and `n_streams` records the batch size. Key design points: - **PTS-anchor batch assembly** — for each batch the element picks the earliest PTS across all non-EOS pads as the *anchor*, then waits up to `max-wait-time` for the other pads to deliver buffers whose PTS lies within `pts-tolerance` of that anchor. Pads that miss the window do not block the batch indefinitely; they simply do not contribute to that cycle. - **Back-pressure on upstream** — each pad has its own bounded queue (`max-queue-size`). When a queue is full, the upstream chain function blocks until the output loop drains it. This prevents unbounded memory growth without dropping frames. - **Late-frame dropping** — if a buffer's PTS is more than `pts-tolerance` *behind* the most recent pushed batch, it is dropped (the batch already moved past that time and there is no way to insert it). - **Per-pad PTS normalization (`sync-mode`)** — when sources have unrelated PTS timelines (e.g. files starting at different timestamps, multiple cameras with independent clocks), the element can normalize PTS before scheduling. See the [Sync Mode](#sync-mode) section. - **Sparse pad indices** — names like `mux.sink_5` work even if `sink_0..sink_4` were never created. Indices must be in `[0, 256)`. > **Important**: each source contributes one buffer per batch, so a downstream > `gvadetect inference-interval=N` (with N > 1) skips every Nth buffer in arrival order, which > means certain source ids will always be skipped. Use `inference-interval=1` (the default) > when the goal is uniform coverage across all input streams. ## How It Works 1. The pipeline requests sink pads (`sink_0`, `sink_1`, ...) — one per input source. Pad index is parsed from the requested name; sparse indices are allowed up to 255. Request a pad without a name to be given the lowest free index. 2. Upstream chain functions push buffers into per-pad queues. If a pad's queue is full, the chain function blocks until the output task drains it (back-pressure to upstream). 3. The output task on the src pad assembles batches in three phases: - **Phase 1 (anchor selection)** — discard any queue head older than the batch already pushed, then pick the earliest valid PTS across all non-EOS pad queue heads as `batch_anchor_pts`. Discarding first keeps the anchor monotonic: a buffer older than the last batch has missed it, and taking it as the next anchor would move batch time (and, in `container` mode, the output buffer's PTS) backwards. - **Phase 2 (wait for contributors)** — wait up to `max-wait-time` for every *active* non-EOS pad to have a head buffer with `|pts - anchor| ≤ pts-tolerance`. Exit early once they all contribute. See [Slow and Failed Sources](#slow-and-failed-sources). - **Phase 3 (collect & push)** — pop one matching buffer per pad, attach `GstAnalyticsBatchMeta`, push downstream in pad-index order. 4. EOS on a sink pad is honored only once that pad's queue is drained: buffers already enqueued before EOS are still batched (so the last frames of a stream, or a single-frame source, are not lost). A pad with no remaining buffers and EOS set is excluded from future batches. When all pads are EOS and all queues are drained, the src pad emits EOS and the task pauses. 5. The muxer presents a single stream assembled from several, so it emits exactly one `stream-start` with its own stream id. Each sink pad's upstream `stream-start` is consumed rather than forwarded. 6. Per-pad `FLUSH_START` / `FLUSH_STOP` events are coalesced — the downstream flush, task pause, queue reset and task restart each happen exactly once per flush cycle no matter how many sink pads receive the events. A flush removes the sticky segment from the `src` pad, so the muxer sends a replacement immediately afterwards. ### Slow and Failed Sources A batch waits only for the sources that are behaving. Each sink pad is in one of three states, and a pad leaves the first one as soon as anything arrives on it — recovery is judged on delivery, not on whether what was delivered could be used. | State | Entered when | Effect | |-------|--------------|--------| | `active` | buffers are arriving and being batched | batches wait for this pad | | `stalled` | nothing has arrived for five batches | batches no longer wait for it | | `late` | its queue head has been discarded as stale for three batches running | batches no longer wait for it | Neither unhealthy state excludes a pad from batching — its buffers still go out whenever they match the anchor. The pad simply stops holding everyone else up, so a dead feed costs its own share of the throughput and nothing more. With four sources at 30 fps and one of them cut off, output goes from 120 to 90 buffers per second rather than collapsing to 74. `stalled` and `late` are separate because they need different responses: nothing is arriving in the first, while in the second the source is alive and working through a backlog that Phase 1 discards until it is current again. The thresholds are counted in batches, not milliseconds, so they scale with `max-wait-time` — already your statement of how long a straggler may hold a batch open. Every change of state is posted on the bus as a `GST_MESSAGE_ELEMENT`, once per transition: ``` gva-streammux-stream-state, pad-index=(uint)3, state=(string)stalled, active-pads=(uint)3, total-pads=(uint)4 ``` `state` is `active`, `stalled` or `late`. Watch for it to find out *which* source went bad without turning on debug logging. ### Adding and Removing Sources at Runtime Sink pads can be requested and released while the pipeline is running. - Releasing a pad that has already reached EOS does **not** end the stream: the muxer only emits EOS once every pad that is still attached has finished. Releasing finished sources one at a time, as a long-running pipeline recycles them, therefore keeps the live ones running. - A pad may be released while its upstream is still pushing. Its chain function is told to stop first, so a source that is still delivering is not left waiting for queue space that will never be freed. - Requesting an index that is already in use is rejected (`request_new_pad` returns `NULL`) rather than replacing the existing pad. Request pads without a name to get the first free index. - If every pad reaches EOS before any of them reported caps, the source caps can never be negotiated. The muxer ends the stream itself, sending stream-start and a segment (and, in `container` mode, the batch caps) ahead of the EOS. - A source that goes quiet, or comes back delivering buffers that are already too old, stops holding up the batches of the sources that are still healthy. See [Slow and Failed Sources](#slow-and-failed-sources). ## Output Modes The output mode is selected explicitly by the `output-mode` property (default `passthrough`); there is no auto-detection from the input media types. The element negotiates its `src` caps **once**, after every live sink pad has reported its caps, and the output loop does not push anything until then. | Mode (`output-mode`) | Output | |----------------------|--------| | **`passthrough`** (default) | Each source buffer is pushed as-is, tagged with a single-stream `GstAnalyticsBatchMeta` (`streams[0].index` = source id). This is the legacy behavior — existing `gvastreammux ! gvadetect` pipelines are unaffected. All sink pads **must negotiate identical caps**: the element compares the pads' caps for equality (it does *not* inspect the media type) and posts an error if they differ. The shared sink caps become the `src` caps. | | **`container`** | Each assembled batch is packed into **one** container buffer with caps `multistream/x-analytics-batch(meta:GstAnalyticsBatchMeta)`. The `GstAnalyticsBatchMeta` holds one `GstAnalyticsBatchStream` per contributing source; each stream carries that source's buffer (`objects[0]`) and its original caps (a `GST_EVENT_CAPS` sticky event). Caps may differ across pads, so this is the mode for **heterogeneous** sources (e.g. video + lidar). Use `gvastreamdemux` downstream to unpack the container back into per-source pads. | > **Important (`container` mode):** a container buffer is *not* raw video, so no video element > (`gvadetect`, `gvawatermark`, …) may follow the mux directly. Insert `gvastreamdemux` to recover > the per-source streams (e.g. video → `gvadetect`, lidar → `g3dinference`) first. > **Caps changes mid-stream (`passthrough` mode):** the `src` caps are fixed at negotiation and > cannot be changed afterwards, because buffers already queued were accepted under the old > caps. If a sink pad reports different caps later — an RTSP source reconnecting at another > resolution, for instance — the element posts an error instead of silently forwarding > mismatched buffers. Use `container` mode for sources whose caps can change. > **Choosing the mode:** use `passthrough` when every source produces the same caps (the common > all-video case) and you want plain buffers downstream. Use `container` whenever the sources are > heterogeneous (mixed media types or differing caps) — a `passthrough` mux with mismatched sink > caps will error out at caps negotiation. `passthrough` mode keeps the historical one-buffer-per-source-per-batch shape; `container` mode is one-buffer-per-batch carrying all sources. ## Properties | Property | Type | Default | Description | |-----------------|----------|-------------|-------------| | `max-fps` | Double | `0` | Output rate cap on the assembled batches (0 = unlimited). Only set for local file sources; setting on RTSP/live sources can stall the pipeline. | | `pts-tolerance` | UInt64 (ns) | `20000000` (20 ms) | Max `\|pts - anchor\|` for a buffer to count as contributing to the current batch. | | `max-wait-time` | UInt64 (ns) | `40000000` (40 ms) | Max time the output task waits for late pads after the anchor is set. After timeout the partial batch is pushed. | | `max-queue-size`| UInt | `2` | Maximum buffers per pad queue. When reached, upstream blocks (back-pressure). | | `sync-mode` | Enum | `none` | How to normalize PTS across pads. See [Sync Mode](#sync-mode). | | `output-mode` | Enum | `passthrough` | `passthrough` (all sink pads must share identical caps) or `container` (one multistream batch buffer per batch). See [Output Modes](#output-modes). | ### Sync Mode Multi-source pipelines often deliver buffers with PTS values that cannot be compared directly (different file starts, multiple cameras with independent clocks, etc.). Without normalization the anchor + tolerance scheduler either deadlocks (a far-future pad never matches the window and back-pressures upstream) or oscillates (anchor jumps each round, batches end up single-stream). `sync-mode` selects the normalization policy. | Mode | What it does | Typical use case | |-------------|------------------------------------------------------------------------------|------------------| | `none` | Use buffer PTS as-is. Default; preserves legacy behavior. | Single source, or upstream already aligned (e.g. `videorate`). | | `first-pts` | Subtract each pad's first observed PTS so every pad starts at 0. Survives a source restarting its timeline (see below). | Multiple file sources whose PTS bases differ; RTSP sources that reconnect. | | `segment` | Subtract each pad's `GST_EVENT_SEGMENT` start. | Standard GStreamer pipelines, including seek/trick mode. | | `pipeline` | Overwrite PTS with `clock.now() - base_time` (pipeline running time). | Single-machine multi-live source where source PTS are unreliable. | | `ntp` | Replace PTS with `GstReferenceTimestampMeta.timestamp` if the buffer carries one. | Cross-device cameras synchronized via NTP/PTP, paired with `rtspsrc add-reference-timestamp-meta=true`. | Per-pad normalization state (first PTS, segment start) is reset on `FLUSH_STOP` and on `PAUSED → READY`, so seek and pipeline restart establish fresh baselines. `first-pts` additionally handles a source whose timeline restarts mid-run, which is what an RTSP feed does when it reconnects: PTS begins from zero again, with `DISCONT` set. The pad's baseline is re-derived so its normalized time **continues from where it left off** rather than restarting at zero — a pad that went back to zero while the others were seconds ahead would have every buffer discarded as a late frame and would never recover. A restart is recognised only when a buffer is flagged `DISCONT` *and* its PTS jumps by more than a second; either signal alone (a dropped frame, a sparse source) leaves the baseline untouched. The segment on the `src` pad follows from the mode. `first-pts`, `segment` and `pipeline` rewrite every PTS onto a zero-based timeline, so the muxer sends a zero-based segment to match. `none` passes PTS through untouched, so it forwards the upstream segment instead — otherwise a downstream sink would read a buffer seeked to T as having a running time of T and wait that long before rendering it. A flush starts a new run, so a seek adopts the new segment. ## Buffer Metadata Each output buffer carries a `GstAnalyticsBatchMeta`. The way it is populated depends on the [output mode](#output-modes): **PASSTHROUGH (video-only):** one buffer per source, single-stream meta. | Field | Type | Description | |------------------------|----------|-------------| | `streams[0].index` | guint | Source pad index this buffer originated from. | | `n_streams` | gsize | Always `1`: a passthrough buffer carries one source. | > Up to and including the 2026.2 release this reported the size of the batch the buffer > belonged to while still filling in only `streams[0]`, so consumers that walk > `streams[0..n_streams)` — `gvametaconvert`, `g3drender`, `g3dobjectfuser` — saw > `batch_size - 1` zeroed entries on every buffer. How many sources were in the batch is a > different fact and is not carried here. **CONTAINER (heterogeneous):** one container buffer per batch, `n_streams` entries. | Field | Type | Description | |-----------------------------|----------|-------------| | `n_streams` | gsize | Number of sources packed in this batch. | | `streams[i].index` | guint | Source pad index of the i-th packed stream. | | `streams[i].objects[0]` | GstBuffer| The i-th source's actual buffer. | | `streams[i].sticky_events` | GstEvent | The i-th source's original caps (a `GST_EVENT_CAPS`). | `gvastreamdemux` reads this meta to route each source's buffer to the matching `src_*` pad. ## Pipeline Examples ### Local files with `max-fps` ```bash gst-launch-1.0 \ gvastreammux name=mux max-fps=30 \ ! queue \ ! gvadetect model=model.xml device=GPU \ pre-process-backend=va-surface-sharing \ ! queue ! gvafpscounter ! fakesink \ filesrc location=video0.h265 ! h265parse ! vah265dec ! mux.sink_0 \ filesrc location=video1.h265 ! h265parse ! vah265dec ! mux.sink_1 ``` ### Multiple files with different PTS bases (`sync-mode=first-pts`) ```bash gst-launch-1.0 \ gvastreammux name=mux sync-mode=first-pts max-fps=30 \ ! queue \ ! gvadetect model=model.xml device=GPU \ ! queue ! gvafpscounter ! fakesink \ filesrc location=clip-recorded-at-10am.mp4 ! qtdemux ! h265parse ! vah265dec ! mux.sink_0 \ filesrc location=clip-recorded-at-2pm.mp4 ! qtdemux ! h265parse ! vah265dec ! mux.sink_1 ``` ### RTSP sources, no `max-fps` ```bash gst-launch-1.0 \ gvastreammux name=mux \ ! queue \ ! gvadetect model=model.xml device=GPU \ pre-process-backend=va-surface-sharing \ ! queue ! gvafpscounter ! fakesink \ rtspsrc location=rtsp://host:8554/stream latency=200 \ ! rtph265depay ! h265parse ! vah265dec ! mux.sink_0 \ rtspsrc location=rtsp://host:8555/stream latency=200 \ ! rtph265depay ! h265parse ! vah265dec ! mux.sink_1 ``` ### Multi-camera NTP/PTP synchronization (`sync-mode=ntp`) When sources are real IP cameras synchronized via NTP (and the SDP advertises an RFC7273 clock), set `rtspsrc add-reference-timestamp-meta=true` so each buffer carries a `GstReferenceTimestampMeta` with absolute time, then let the mux use that as PTS: ```bash gst-launch-1.0 \ gvastreammux name=mux sync-mode=ntp pts-tolerance=33000000 \ ! queue ! gvadetect model=model.xml device=GPU \ ! queue ! gvafpscounter ! fakesink \ rtspsrc location=rtsp://cam1 ntp-sync=true add-reference-timestamp-meta=true \ ! rtph265depay ! h265parse ! avdec_h265 ! mux.sink_0 \ rtspsrc location=rtsp://cam2 ntp-sync=true add-reference-timestamp-meta=true \ ! rtph265depay ! h265parse ! avdec_h265 ! mux.sink_1 ``` `pts-tolerance` is widened above the typical inter-frame interval to accommodate per-camera network jitter; tune it for your environment. ### Pair with `gvastreamdemux` for per-source output ```bash gst-launch-1.0 \ gvastreammux name=mux max-fps=30 \ ! queue \ ! gvadetect model=model.xml device=GPU \ ! gvastreamdemux name=demux \ demux.src_0 ! queue ! gvafpscounter ! fakesink \ demux.src_1 ! queue ! gvafpscounter ! fakesink \ filesrc location=video0.h265 ! h265parse ! vah265dec ! mux.sink_0 \ filesrc location=video1.h265 ! h265parse ! vah265dec ! mux.sink_1 ``` ### Heterogeneous sources: video + lidar (CONTAINER mode) Heterogeneous sources (here lidar via `g3dlidarparse` alongside video) require [CONTAINER mode](#output-modes), selected with `output-mode=container`. The container batch must be unpacked by `gvastreamdemux` before any per-stream processing — no `gvadetect` may sit between the mux and demux. In this example each demuxed branch goes straight to `fakesink` (`src_0`/`src_1` = video, `src_2` = lidar): ```bash gst-launch-1.0 -e \ gvastreammux name=mux output-mode=container sync-mode=first-pts \ ! queue \ ! gvastreamdemux name=demux \ demux.src_0 ! queue ! gvafpscounter ! fakesink \ demux.src_1 ! queue ! gvafpscounter ! fakesink \ demux.src_2 ! queue ! fakesink \ filesrc location=video0.h265 ! h265parse ! vah265dec ! mux.sink_0 \ filesrc location=video1.h265 ! h265parse ! vah265dec ! mux.sink_1 \ multifilesrc location=velodyne/%06d.bin start-index=0 caps=application/octet-stream \ ! g3dlidarparse frame-rate=10 ! mux.sink_2 ``` > Lidar typically runs at a lower rate (e.g. 10 fps) than video (e.g. 30 fps). With aligned PTS, > batches that have a coincident lidar frame carry it (`n_streams` includes the lidar pad); > video-only intervals produce video-only batches. Ensure the lidar branch produces valid PTS > (`g3dlidarparse frame-rate=...`) and pick a `sync-mode` so the two timelines are comparable. ## Tuning Notes - `pts-tolerance` should comfortably accommodate the largest expected inter-source PTS drift *after* normalization. For 30 fps sources a value of 16–33 ms is a reasonable starting point. - `max-wait-time` bounds the worst-case batch latency when one pad stops delivering. Set it somewhat above one frame interval (e.g. 40–50 ms for 30 fps) so transient stalls do not immediately break the batch. - `max-queue-size` controls the back-pressure depth. Larger values absorb more upstream bursts at the cost of memory; 2–4 is usually enough. ## Element Details (gst-inspect-1.0) ``` Factory Details: Long-name GVA Stream Muxer Klass Muxer Description Muxes multiple streams with PTS-based synchronization. The 'output-mode' property selects between PASSTHROUGH (each source buffer pushed as-is; all sink pads must share identical caps) and CONTAINER (each batch packed into one multistream/x-analytics-batch buffer, e.g. video + lidar). Author Intel Corporation Pad Templates: SINK template: 'sink_%u' (On request, video/x-raw, video/x-raw(memory:VAMemory) NV12, video/x-raw(memory:DMABuf) DMA_DRM, application/x-lidar, application/x-radar-processed; on Windows video/x-raw(memory:D3D11Memory) and video/x-raw(memory:D3D12Memory) NV12/BGRA instead of VAMemory and DMABuf) SRC template: 'src' (Always, the sink caps above (passthrough) OR multistream/x-analytics-batch(meta:GstAnalyticsBatchMeta) (container)) Element Properties: max-fps Double, range 0-Inf, default 0 pts-tolerance UInt64 ns, default 20000000 (20 ms) max-wait-time UInt64 ns, default 40000000 (40 ms) max-queue-size UInt, range 1-..., default 2 sync-mode Enum (none, first-pts, segment, pipeline, ntp), default none output-mode Enum (passthrough, container), default passthrough ``` Run `gst-inspect-1.0 gvastreammux` against your installation for the authoritative output.