# Converting NVIDIA DeepStream Pipelines to Deep Learning Streamer Pipeline Framework This article explains how to convert a pipeline from NVIDIA DeepStream to the Deep Learning Streamer Pipeline Framework. A working example is provided at each step to help illustrate the required modifications. > **Note:** The intermediate steps of the pipeline are not meant to run. They are > simply there as a reference example of the changes being made in each > section. ## Contents - [Preparing Your Model](#preparing-your-model) - [Conversion Examples](#conversion-examples) - [Command Line Applications](#command-line-applications) - [Python Applications](#python-applications) - [Conversion Rules](#conversion-rules-for-pipeline-elements) - [Mux and Demux Elements](#mux-and-demux-elements) - [Inferencing Elements](#inferencing-elements) - [Video Processing Elements](#video-processing-elements) - [Metadata Elements](#metadata-elements) - [Multiple Input Streams](#multiple-input-streams) - [DeepStream to DL Streamer Mapping](#deepstream-to-dl-streamer-mapping) - [Element Mapping](#element-mapping) - [Property Mapping](#property-mapping) - [AI-Supported Conversions](#ai-supported-conversions) ## Preparing Your Model > **Note:** To use the Deep Learning Streamer Pipeline Framework and the OpenVINO™ Toolkit, the > model needs to be in Intermediate Representation (IR) format. To convert > your model to this format, follow the steps in [model preparation](./model_preparation.md). > ## Conversion Examples ### Command Line Applications The following sections show how to convert a DeepStream pipeline to a DL Streamer pipeline. The DeepStream pipeline is taken from one of the [examples](https://github.com/NVIDIA-AI-IOT/deepstream_reference_apps). It reads a video stream from the input file, decodes it, runs inference, overlays the inference results on the video, re-encodes it, and outputs a new .mp4 file. ```shell filesrc location=input_file.mp4 ! decodebin ! \ nvstreammux batch-size=1 width=1920 height=1080 ! queue ! \ nvinfer config-file-path=./config.txt ! \ nvvideoconvert ! "video/x-raw(memory:NVMM), format=RGBA" ! \ nvdsosd ! queue ! \ nvvideoconvert ! "video/x-raw, format=I420" ! videoconvert ! avenc_mpeg4 bitrate=8000000 ! qtmux ! filesink location=output_file.mp4 ``` The mapping below represents the typical changes that need to be made to the pipeline to convert it to Deep Learning Streamer Pipeline Framework. The pipeline is broken down into sections based on the elements used in the pipeline. ![image](./_assets/deepstream_mapping_dlstreamer.png) ### Python Applications While the GStreamer command line allows a quick demonstration of a running pipeline, fine-grained control typically involves using a GStreamer pipeline object programmatically in Python or C/C++. This section illustrates how to convert a [DeepStream Python example](https://github.com/NVIDIA-AI-IOT/deepstream_python_apps/tree/master/apps/deepstream-test1) into a [DL Streamer Python example](https://github.com/open-edge-platform/dlstreamer/tree/main/samples/gstreamer/python/hello_dlstreamer). Both applications implement the same functionality, but they use DeepStream and DL Streamer elements differently, as illustrated in the table below. The elements in __bold__ are vendor-specific, while the others are regular GStreamer elements. | DeepStream Element | DL Streamer Element | Function | |---|---|---| | filesrc | filesrc | Read video file | | h264parse ! __nvv4l2decoder__ | decodebin3 | Decode video file | | __nvstreammux__ ! __nvinfer__ | __gvadetect__ ! queue | Create batch buffer and run AI inference | | __nvvideoconvert__ | videoconvertscale | Convert video format | | __nvosd__ | __gvawatermark__ | Overlay analytics results | | __nv3dsink__ | audiovideosink | Render results in a window | Such pipelines can be created programmatically in a Python application using a sequence of API calls: ```python element = Gst.ElementFactory.make(...) element.set_property(...) pipeline.add(element) element.link(next_element) ``` Please note that DeepStream and DL Streamer applications use the same set of regular GStreamer library functions to construct pipelines. The difference lies in which elements are created and linked. In addition, the DL Streamer `decodebin3` element uses late linking within a callback function.
DeepStream Pipeline Creation in Python DL Streamer Pipeline Creation in Python 

pipeline = Gst.Pipeline()
...
source = Gst.ElementFactory.make("filesrc", "file-source")
h264parser = Gst.ElementFactory.make("h264parse", "h264-parser")
...
source.set_property('location', args[1])
streammux.set_property('batch-size', 1)
...
pipeline.add(source)
pipeline.add(h264parser)
...
source.link(h264parser)
h264parser.link(decoder)


pipeline = Gst.Pipeline()
...
source = Gst.ElementFactory.make("filesrc", "file-source")
decoder = Gst.ElementFactory.make("decodebin3", "media-decoder")
...
source.set_property('location', args[1])
detect.set_property('batch-size', 1)
...
pipeline.add(source)
pipeline.add(decoder)
...
source.link(decoder)
decoder.connect("pad-added",
  lambda element, pad, data: element.link(data) if pad.get_name().find("video") != -1 and not pad.is_linked() else None,
  detect)
Once the pipelines are created, both applications register a custom probe handler and attach it to the sink pad of the overlay element. The probe registration code is again very similar, except that different elements are used: `nvosd` and `gvawatermark`.
DeepStream Probe Registration DL Streamer Probe Registration
osdsinkpad = nvosd.get_static_pad("sink") osdsinkpad.add_probe(Gst.PadProbeType.BUFFER, osd_sink_pad_buffer_probe, 0) watermarksinkpad = watermark.get_static_pad("sink") watermarksinkpad.add_probe(Gst.PadProbeType.BUFFER, watermark_sink_pad_buffer_probe, 0)
The probe function iterates over the prediction metadata produced by the AI model. Here, the DeepStream and DL Streamer implementations differ significantly. The DeepStream sample uses DeepStream-specific structures for batches of frames, individual frames, and objects within a frame. In contrast, the DL Streamer sample uses regular GStreamer data structures from the [GstAnalytics metadata library](https://gstreamer.freedesktop.org/documentation/analytics/index.html?gi-language=python#analytics-metadata-library). Please also note that the DL Streamer handler runs once per frame, while the DeepStream sample runs once per batch of frames.
DeepStream Probe Implementation DL Streamer Probe Implementation 

batch_meta = pyds.gst_buffer_get_nvds_batch_meta(hash(gst_buffer))
...
l_frame = batch_meta.frame_meta_list
...
while l_frame is not None:
  frame_meta = pyds.NvDsFrameMeta.cast(l_frame.data)
  ...
  l_obj=frame_meta.obj_meta_list
    while l_obj is not None:
      ... process object metadata


# no batch meta in DL Streamer, probes run per-frame
...
frame_meta = GstAnalytics.buffer_get_analytics_relation_meta(buffer)
for obj in frame_meta:
  ... process object metadata
The last table compares the pipeline execution logic. Both applications set the pipeline state to `PLAYING` and run the main GStreamer event loop. The DeepStream sample invokes a predefined event loop from a DeepStream library, while the DL Streamer application explicitly adds the message processing loop. Both implementations keep running the pipeline until an end-of-stream message is received.
DeepStream Pipeline Execution DL Streamer Pipeline Execution 

# create an event loop and feed gstreamer bus messages to it
loop = GLib.MainLoop()
bus = pipeline.get_bus()
bus.add_signal_watch()
bus.connect ("message", bus_call, loop)
 
# start play back and listen to events
pipeline.set_state(Gst.State.PLAYING)
  try:
    loop.run()
  except:
    pass
 
pipeline.set_state(Gst.State.NULL)


bus = pipeline.get_bus()
 
# run explicit pipeline loop, handle ERROR and EOS messages
pipeline.set_state(Gst.State.PLAYING)
terminate = False
while not terminate:
  msg = bus.timed_pop_filtered(Gst.CLOCK_TIME_NONE, Gst.MessageType.EOS | Gst.MessageType.ERROR)
  if msg:
    if msg.type == Gst.MessageType.ERROR:
      ... handle errors
      terminate = True
    if msg.type == Gst.MessageType.EOS:
      terminate = True
pipeline.set_state(Gst.State.NULL)
## Conversion Rules for Pipeline Elements ### Mux and Demux Elements The following sections provide more details on how to replace each element. - Remove `nvstreammux` and `nvstreamdemux` and all their properties. - These elements combine multiple input streams into a single batched video stream (DeepStream-specific). Deep Learning Streamer takes a different approach: it employs a generic GStreamer syntax to define parallel streams. The cross-stream batching happens at the inferencing elements by setting the same `model-instance-id` property. - In this example, there is only one video stream, so it can be skipped for now. See the [Multiple Input Streams](#multiple-input-streams) section below for more information on how to construct multi-stream pipelines. At this stage, you have removed `nvstreammux` and the `queue` that followed it. Keep in mind that the `batch-size` property is also removed. It will be added in the next section as a property of the Pipeline Framework inference elements. ```shell filesrc location=input_file.mp4 ! decodebin3 ! \ nvinfer config-file-path=./config.txt ! \ nvvideoconvert ! "video/x-raw(memory:NVMM), format=RGBA" ! \ nvdsosd ! queue ! \ nvvideoconvert ! "video/x-raw, format=I420" ! videoconvert ! avenc_mpeg4 bitrate=8000000 ! qtmux ! filesink location=output_file.mp4 ``` ### Inferencing Elements - Remove `nvinfer` and replace it with `gvainference`, `gvadetect` or `gvaclassify` depending on the following use cases: - For detection of objects on full frames and outputting a region of interest, use [gvadetect](../elements/gvadetect.md). This replaces `nvinfer` when it is used in primary mode. - Replace the `config-file-path` property with `model` and `gvadetect` properties. - `gvadetect` generates `GstVideoRegionOfInterestMeta`. - For classification of previously detected objects, use [gvaclassify](../elements/gvaclassify.md). This replaces `nvinfer` when it is used in secondary mode. - Replace the `config-file-path` property with `model` and `gvadetect` properties. - `gvaclassify` requires `GstVideoRegionOfInterestMeta` as input. - For generic full frame inference, use [gvainference](../elements/gvainference.md). This replaces `nvinfer` when used in primary mode. - `gvainference` generates `GstGVATensorMeta`. In this example, you will use `gvadetect` to infer on the full frame and output the regions of interest. `batch-size` is also added for consistency with what was removed above (the default value is `1` so it is not needed). The `config-file-path` property is replaced with `model` and and [Model Info Section](./model_info_xml.md). ```shell filesrc location=input_file.mp4 ! decodebin3 ! \ gvadetect model=./model.xml batch-size=1 ! queue ! \ nvvideoconvert ! "video/x-raw(memory:NVMM), format=RGBA" ! \ nvdsosd ! queue ! \ nvvideoconvert ! "video/x-raw, format=I420" ! videoconvert ! avenc_mpeg4 bitrate=8000000 ! qtmux ! filesink location=output_file.mp4 ``` ### Video Processing Elements - Replace NVIDIA-specific video processing elements with native GStreamer elements. - `nvvideoconvert` with `vapostproc` (GPU) or `videoconvert` (CPU). If the `nvvideoconvert` is being used to convert to/from `memory:NVMM` it can be removed. - `nvv4ldecoder` can be replaced with `va{CODEC}dec`, for example `vah264dec` for decoding only. Alternatively, the native `decodebin3` GStreamer element can be used to automatically choose an available decoder. - Some caps filters that follow an inferencing element may need to be adjusted or removed. Pipeline Framework inferencing elements do not support color space conversion in post-processing. You will need to have a `vapostproc` or `videoconvert` element to handle this. In the example below, you remove a few caps filters and instances of `nvvideoconvert` used for conversions from DeepStream's NVMM because Pipeline Framework uses standard GStreamer structures and memory types. You use the standard `videoconvert` GStreamer element to do color space conversion on CPU. However, if available, it is recommended to use `vapostproc` to run on Intel Graphics. Also, you use the standard `decodebin` GStreamer element to choose an appropriate demuxer and decoder depending on the input stream as well as what is available on the system. ```shell filesrc location=input_file.mp4 ! decodebin3 ! \ gvadetect model=./model.xml batch-size=1 ! queue ! \ nvdsosd ! queue ! \ videoconvert ! avenc_mpeg4 bitrate=8000000 ! qtmux ! filesink location=output_file.mp4 ``` ### Metadata Elements - Replace `nvtracker` with [gvatrack](../elements/gvatrack.md). - Remove the `ll-lib-file` property. Optionally, replace it with `tracking-type` if you want to specify the used algorithm. By default, it will use the 'short-term' tracker. - Remove all other properties. - Replace `nvdsosd` with [gvawatermark](../elements/gvawatermark.md). - Remove all properties. - Replace `nvmsgconv` with [gvametaconvert](../elements/gvametaconvert.md). - `gvametaconvert` can be used to convert metadata from inference elements to JSON and to output metadata to the GST_DEBUG log. - It has optional properties to configure what information goes into the JSON object including frame data for frames with no detections found, tensor data, the source of the inferences, and tags, and a user-defined JSON object that is attached to each output for additional custom data. - Replace `nvmsgbroker` with [gvametapublish](../elements/gvametapublish.md). - `gvametapublish` can be used to output the JSON messages, generated by `gvametaconvert` to stdout, a file, MQTT or Kafka. - **Key difference:** DeepStream's `nvmsgbroker` is a **sink** element (requires a `tee` to split the stream), but DL Streamer's `gvametapublish` is a **pass-through transform** — it publishes metadata and forwards buffers downstream. No `tee` is needed; place it inline before the display/encode sink. The only metadata processing done in this pipeline serves to overlay the inference results on the video, for which you use `gvawatermark`. ```shell filesrc location=input_file.mp4 ! decodebin3 ! \ gvadetect model=./model.xml batch-size=1 ! queue ! \ gvawatermark ! queue ! \ videoconvert ! avenc_mpeg4 bitrate=8000000 ! qtmux ! filesink location=output_file.mp4 ``` ### Multiple Input Streams Unlike DeepStream, where all sources need to be linked to the sink pads of the `nvstreammux` element, Pipeline Framework uses existing GStreamer mechanisms to define multiple parallel video processing streams. This approach allows reusing native GStreamer elements within the pipeline. The input streams can share the same Inference Engine if they have the same `model-instance-id` property. This enables creating inference batching across streams. For DeepStream, a simple pipeline involving two streams is presented in the code snippet below. - The first line defines a common inference element for two (muxed and batched) streams. - The second line defines per-stream input operations prior to muxing. - The third line defines per-stream output operations after de-muxing. ```bash nvstreammux ! nvinfer batch-size=2 config-file-path=./config.txt ! nvstreamdemux \ filesrc ! decode ! mux.sink_0 filesrc ! decode ! mux.sink_1 \ demux.src_0 ! encode ! filesink demux.src_1 ! encode ! filesink ``` The native GStreamer syntax below uses the Pipeline Framework to define operations for two parallel streams. When you set `model-instance-id` to the same value, both streams will share the same instance of the `gvadetect` element. That means you can define the shared inference parameters (the model, the batch size, ...) only in the first line: ```shell filesrc ! decode ! gvadetect model-instance-id=model1 model=./model.xml batch-size=2 ! encode ! filesink \ filesrc ! decode ! gvadetect model-instance-id=model1 ! encode ! filesink ``` ## DeepStream to DL Streamer Mapping ### Element Mapping The table below provides a quick reference for mapping typical DeepStream elements to Deep Learning Streamer elements or GStreamer. | DeepStream Element | DL Streamer Element | |---|---| | [nvinfer](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html) | [gvadetect](../elements/gvadetect), [gvaclassify](../elements/gvaclassify.md), [gvainference](../elements/gvainference.md) | | [nvdsosd](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvdsosd.html) | [gvawatermark](../elements/gvawatermark.md) | | [nvtracker](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvtracker.html) | [gvatrack](../elements/gvatrack.md) | | [nvmsgconv](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvmsgconv.html) | [gvametaconvert](../elements/gvametaconvert.md) | | [nvmsgbroker](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvmsgbroker.html) | [gvametapublish](../elements/gvametapublish.md) | | DeepStream Element | GStreamer Element | |---|---| | nvvideoconvert | videoconvert | | nvv4l2decoder | decodebin3 | | nvv4l2h264dec | vah264dec | | nvv4l2h265dec | vah265dec | | nvv4l2h264enc | vah264enc | | nvv4l2h265enc | vah265enc | | nvv4l2vp8dec | vavp8dec | | nvv4l2vp9dec | vavp9dec | | nvv4l2vp8enc | vavp8enc | | nvv4l2vp9enc | vavp9enc | ## Property Mapping NVIDIA DeepStream uses a combination of model configuration files and DeepStream element properties to specify inference actions, as well as pre- and post-processing steps before/after running inference, as documented [here](https://docs.nvidia.com/metropolis/deepstream/dev-guide/text/DS_plugin_gst-nvinfer.html). Similarly, the Deep Learning Streamer Pipeline Framework uses GStreamer element properties for inference settings and [Model Info Section](./model_info_xml.md) in model's XML file for pre- and post-processing steps. The following table shows how to map commonly used NVIDIA DeepStream configuration properties to Deep Learning Streamer settings. | NVIDIA DeepStream config file | NVIDIA DeepStream element property | Deep Learning Streamer model info section | Deep Learning Streamer element property | Description | |---|---|---|---|---| | model-engine-file | model-engine-file |   | model | The path to an inference model network file. | | labelfile-path |   |   | labels-file | The path to a .txt file containing object classes. | | network-type <0..3> |   |   |
gvadetect for detection, instance segmentation
gvaclassify for classification, semantic segmentation

| Type of inference operation. | | batch-size | batch-size |   | batch-size | The number of frames batched together for a single inference. | | maintain-aspect-ratio |   | resize_type: fit_to_window |   | The number of frames batched together for a single inference. | | num-detected-classes |   |   |   | The number of classes detected by the model, inferred from a label file by Deep Learning Streamer. | | interval | interval |   | inference-interval | An inference action executed every Nth frame. Note that Deep Learning Streamer value is greater by 1. | |   | threshold |   | threshold | The threshold for detection results. |
## AI-Supported Conversions In addition to the detailed description of the conversion rules between DeepStream and Deep Learning Streamer applications presented above, we also provide the ability to perform such conversions automatically using a coding agent. As part of the [DL Streamer](https://github.com/open-edge-platform/dlstreamer) project, guidelines for the coding agent have been prepared to support conversion of applications written in **Python**; see [deepstream-python-conversion](https://github.com/open-edge-platform/dlstreamer/blob/main/.github/skills/dlstreamer-coding-agent/example-prompts/deepstream-python-conversion.md). They also support conversion of **C/C++** applications; see [deepstream-cpp-conversion](https://github.com/open-edge-platform/dlstreamer/blob/main/.github/skills/dlstreamer-coding-agent/example-prompts/deepstream-cpp-conversion.md). To see examples of DeepStream applications automatically converted by the coding agent, refer to the [auto_generated_samples](https://github.com/open-edge-platform/dlstreamer/tree/main/samples/auto_generated_samples) directory, which includes [Python](https://github.com/open-edge-platform/dlstreamer/tree/main/samples/auto_generated_samples/deepstream_python_conversion) and [C++](https://github.com/open-edge-platform/dlstreamer/tree/main/samples/auto_generated_samples/deepstream_cpp_conversion) conversion examples. ### Supported Languages The coding agent supports conversion of DeepStream applications developed in the following programming languages: - **Python** - applications using Python bindings for DeepStream. The output code is based on the [python-app-template](https://github.com/open-edge-platform/dlstreamer/tree/main/.github/skills/dlstreamer-coding-agent/assets/python-app-template.py), which demonstrates the structure of a DL Streamer application in Python using Sample Pipeline Processing. - **C/C++** - native DeepStream applications written in C or C++. The output code is based on the [cpp-app-template](https://github.com/open-edge-platform/dlstreamer/tree/main/.github/skills/dlstreamer-coding-agent/assets/cpp-app-template.cpp), which demonstrates the structure of a DL Streamer application in C++ using Sample Pipeline Processing. > **Note:** By default, the coding agent generates the converted application in the same programming language as the source DeepStream application. However, this behavior can be changed by modifying the conversion prompt to explicitly request a different target language. See also: [Coding Agent](./coding_agent.md)