Model Preparation#

When getting started with Deep Learning Streamer, the recommended way to obtain models ready for use in video analytics pipelines is to use the per-source conversion scripts under scripts/download_models (download_hf_models.py for Hugging Face, download_ultralytics_models.py for Ultralytics YOLO, download_timm_models.py for TIMM, and download_other_models.py for a handful of other helper models). See Download and Convert Models and the script’s own README for details.

This way, you will be able to easily perform the most popular tasks, such as object detection and classification, instance segmentation, face localization and many others. For examples of how to set up Deep Learning Streamer pipelines that carry out these functions, refer to the sample directory.

If you are interested in designing custom pipelines, make sure to review the Supported Models table for guidance on whether Deep Learning Streamer elements require specific configurations (defined by the model-proc or labels files) for your selected model.

1. Model file format used by OpenVINO™ Toolkit#

Video Analytics GStreamer plugins utilize Intel® Distribution of OpenVINO™ Toolkit as a back-end for high efficiency-inference on Intel® CPU and accelerators (GPU, NPU, FPGA) and require the model to be converted from the training framework format (e.g., TensorFlow or Caffe) into a format optimized for inference on the target device. The model format used by OpenVINO™ Toolkit is called Intermediate Representation (IR) and consists of two files:

  • xml - a file with the description of network topology

  • bin: - a binary file (potentially big) with model weights

You can either:

  1. Choose a pre-trained model that is already available in the OpenVINO IR format.

  2. Use OpenVINO™ Toolkit Model Conversion method for converting your model from the training framework format (e.g., TensorFlow) to the IR format.

When converting a custom model, you can optionally utilize the Post-Training Model Optimization and Compression for converting the model into a performance efficient, and more hardware-friendly representation. For example you can quantize it from 32-bit floating point-precision into 8-bit integer precision. This gives a significant performance boost (up to 4x) on some hardware platforms including the CPU, with only a minor accuracy drop.

2. Model pre- and post-processing#

Deep Learning Streamer plugins are capable of optionally performing certain pre- and post-processing operations before/after inference.

Pre- and post-processing can be configured using:

  • The model-proc file (mostly applicable to models from Open Model Zoo). Its format and all possible pre- and post-processing configuration parameters are described in the model-proc description page.

  • The “model_info” section inside the .xml file of OpenVINO™ Intermediate Representation with network topology, is described on the model_info description page.

Both methods yield the same results, but the “model_info” is recommended as it simplifies dynamic deployments (like re-training models with Intel® Geti™) as certain model pre- and post-processing parameters are tightly coupled with model training results.

Pre-processing is an input data transformation into an appropriate form which a neural network expects it to be. Most pre-processing actions are designed for images. Pipeline Framework provides several pre-processing back-ends depending on your use case.

To use one of them, set the pre-process-backend property of the inference element to one from the table below.

Default behavior: If the property is not set, Pipeline Framework will pick ie if the system memory is used in the pipeline, va - for GPU memory (VAMemory and DMABuf). On Windows, d3d12-surface-sharing is picked for D3D12Memory and d3d11-surface-sharing for D3D11Memory when the inference device is a GPU; d3d12 and d3d11 are picked for NPU, AUTO and MULTI devices, or when surface sharing is not supported with the current settings (see Windows pre-processing backends).

pre-process-backend

Description

Memory type in pipeline

Configurable with model-proc?

ie

Short for “Inference Engine”. It resizes an image with a bilinear algorithm and sets color format which is deduced from current media. All that’s done with capabilities provided by Inference Engine from OpenVINO™ Toolkit.

System

No

opencv

All power of OpenCV is leveraged for input image pre-processing. Provides a wide variety of operations that can be performed on image.

System

Yes


va

Should be used in pipelines with GPU memory. Performs mapping to the system memory and uses VA pre-processor.


VAMemory
and
DMABuf

Yes


va-surface-sharing

Should be used in pipelines with GPU memory and GPU inference device. Doesn’t perform mapping to the system memory. As a pre-processor, it performs image resize, crop, and sets color format to NV12.


VAMemory

Partially

Windows pre-processing backends#

On Windows, the VA backends are replaced by Direct3D backends. ie and opencv are available for system memory, as on Linux.

pre-process-backend

Description

Memory type in pipeline

d3d11

Resizes, crops and converts the color format on the GPU with the GStreamer D3D11 converter, then maps the result to system memory for inference. Works with GPU and NPU inference devices.

D3D11Memory

d3d11-surface-sharing

Converts the frame on the GPU into a BGRA/RGBA texture at model input size and passes it to OpenVINO™ through OpenCL/D3D11 interop. The frame is not copied to system memory. GPU inference device only.

D3D11Memory

d3d12

Same as d3d11, using the GStreamer D3D12 converter. Works with GPU and NPU inference devices.

D3D12Memory

d3d12-surface-sharing

Converts the frame on the GPU into a BGRA/RGBA texture at model input size and copies it into a shared D3D12 buffer. OpenCL imports the buffer through cl_khr_external_memory, so the frame is not copied to system memory. GPU inference device only.

D3D12Memory

Notes:

  • None of the D3D backends support custom pre-processing libraries (custom-preproc-lib).

  • If a surface sharing backend is not supported with the current settings (for example, device=NPU), the element falls back to d3d11 or d3d12 and logs a warning.

  • Surface sharing backends hand OpenVINO™ a BGRA or RGBA tensor that matches the channel order in the model’s rt_info. If the model specifies no channel order, BGR is assumed.

  • If pre-process-backend is not set, inference elements accept both D3D11Memory and D3D12Memory. D3D12 decoders (used by decodebin3 on Windows 10 and later) then output D3D12Memory. A d3d11 backend accepts only D3D11Memory, and a d3d12 backend accepts only D3D12Memory.

Post-processing uses so called converters to process raw inference results. The converters transform the results from raw output tensors to a GStreamer representation attached to a frame as metadata.

If there is no suitable pre- and/or post-processing implementation in the Deep Learning Streamer, Custom Processing can be used.

For models with custom operations not natively supported by OpenVINO™, see OpenVINO Custom Operations.

3. Specify model files in GStreamer elements#

The path to the .xml model file must be specified by the mandatory model property of GStreamer inference elements - gvainference/gvadetect/gvaclassify. Below is an example of a pipeline with object detection (gvadetect) and classification (gvaclassify)

gvadetect model=MODEL1_FILE_PATH.xml ! gvaclassify model=MODEL1_FILE_PATH.xml

The .bin file is expected to be located in the same folder as the .xml file, and to have the same filename (with a different extension).

Below is the example of a pipeline including gvadetect and gvaclassify with pre-processing/post-processing rules:

gvadetect model=MODEL1_FILE_PATH.xml model-proc=MODEL1_FILE_PATH.json ! gvaclassify model=MODEL2_FILE_PATH.xml model-proc=MODEL2_FILE_PATH.json