g3dbevfusion#

Runs BEVFusion camera + LiDAR 3D object detection on a gvastreammux batch of six surround-view cameras and one LiDAR, and attaches the detections to the LiDAR stream.

Overview#

The g3dbevfusion element is a GstBaseTransform that runs in place on the container batch (multistream/x-analytics-batch) produced by gvastreammux output-mode=container. It runs the OpenVINO port of BEVFusion from openvino_contrib (modules/openvino_bevfusion), which reproduces the mmdetection3d BEVFusion lidar-camera model trained on nuScenes.

The model runs in one of two modes. The element picks the mode for each batch from the streams it carries:

Sensors declared in calibration

Streams in the batch

Mode for this batch

LiDAR + 6 cameras

LiDAR and all 6 cameras

camera + LiDAR fusion

LiDAR + 6 cameras

LiDAR, some cameras missing

LiDAR-only, with a warning on the first occurrence

LiDAR only

LiDAR

LiDAR-only

any

no LiDAR

passed through without detections, with a warning on the first occurrence

BEVFusion has no camera-only mode: its camera branch takes a depth map projected from the LiDAR points. Declaring between 1 and 5 cameras is an error at startup, because the camera branch is exported for exactly six. A summary of how many frames ran in each mode is logged at INFO when the element stops.

Detections are attached as GstAnalytics3DODMtd to the LiDAR buffer inside the batch, like g3dinference does. A sticky g3d/calibration event carries every camera’s projection and the class names downstream, so g3drender can draw the boxes on each camera image and colour them per class.

Properties#

Property

Type

Default

Description

config

String

NULL

Runtime JSON describing the model files (see Runtime configuration). Required.

calibration

String

NULL

JSON naming the mux pad of the LiDAR and of each camera, with each camera’s calibration, and optionally a per-frame table (see Calibration JSON schema). Required.

device

String

GPU

Device for the dense networks: camera encoder, geometry and detection head. The custom voxelize and BEV-pool ops always run on the GPU.

score-threshold

Float [0, 1]

0.3

Minimum detection score to attach. The model proposes 200 boxes per frame; most score close to 0.

Preparing the model#

samples/gstreamer/gst_launch/g3dbevfusion/g3dbevfusion_prepare_model.sh <openvino-version> prepares bevfusion_ov_model_<openvino-version>/, everything the element loads, from openvino_contrib’s openvino_bevfusion module:

  1. creates a temporary Python virtual environment with CPU PyTorch, NumPy and openvino==<openvino-version>;

  2. builds the three custom-op extension libraries against that openvino wheel;

  3. downloads the published mmdetection3d BEVFusion lidar-camera checkpoint and strips it to its weights;

  4. exports the OpenVINO IRs with export.py;

  5. copies the IRs, the extension libraries and the GPU kernels into the output directory and writes the runtime configuration g3dbevfusion.json there.

The virtual environment, sources, checkpoint and build trees are removed at the end. g3dbevfusion_prepare_data.sh downloads nuScenes v1.0-mini and converts it into nuscenes_mini_seq_12Hz/ (see below).

sudo apt-get install git curl g++ python3-venv opencl-c-headers ocl-icd-opencl-dev
./g3dbevfusion_prepare_model.sh 2026.5.0
./g3dbevfusion_prepare_data.sh

Pass the full OpenVINO version of the DL Streamer release that runs the pipeline. It must be 2026.5.0 or later: on earlier releases the OpenVINO GPU plugin miscomputes the detection head and fusion detects nothing, so the script refuses them. The extensions only load into the OpenVINO release they were built against. For a DL Streamer built on an OpenVINO nightly, pass the matching pre-release wheel version, for example 2026.5.0.dev20261030, with PIP_EXTRA_INDEX_URL=https://storage.openvinotoolkit.org/simple/wheels/nightly. The export compiles the custom ops on the GPU, so in a container the GPU has to be available (--device /dev/dri).

Run either script with --help for every option, including BEVFUSION_ROOT to use an existing openvino_contrib checkout and NUSCENES_ROOT to convert an existing nuScenes copy.

The model script checks out a fixed commit of openvinotoolkit/openvino_contrib and applies the patches in samples/gstreamer/gst_launch/g3dbevfusion/patches/ on top of it. They carry the openvino_bevfusion fixes that g3dbevfusion relies on, from openvino_contrib#1197, until that PR is merged (override the source with OPENVINO_CONTRIB_URL and OPENVINO_CONTRIB_REF). They also include a fix that gives the BEV-pool op a zeroed accumulator: without it the camera branch pools into stale GPU memory and fusion detects nothing on OpenVINO 2026.5. Two more patches speed up the model without changing its results: one removes a per-frame allocation that took about half of the sparse encoder’s time, and one exports the camera backbone without the 5-D tensors that the GPU plugin runs slowly. Another raises the limits of the LiDAR branch to 350,000 points and 160,000 voxels, room for the ten LiDAR sweeps the model is trained on. The element reads both limits from the exported model. A model prepared without that patch takes 70,000 points, about one sweep; the element drops the points beyond that, with a warning. The last patch makes the sparse encoder’s kernel choice reproducible: it was made by timing, between kernels that give slightly different results. Detections can still differ slightly from run to run, because voxelization picks the points it averages in GPU-scheduling order.

Runtime configuration#

{
  "model_dir": ".",
  "extensions": [
    "extensions/libopenvino_bevpool_extension.so",
    "extensions/libopenvino_voxelize_extension.so",
    "extensions/libopenvino_sparse_encoder_extension.so"
  ],
  "gpu_custom_layers": "gpu_kernels/gpu_custom_layers.xml"
}

This is the file g3dbevfusion_prepare_model.sh writes. Relative paths are resolved against the configuration file’s directory. The sparse encoder library reads its OpenCL kernel, sparse_conv.cl, from its own directory, and the kernels named in gpu_custom_layers.xml are resolved relative to that file. At startup the element checks model_dir/config.json (written by export.py) against the image size, feature size, camera count and class count it was built for, and fails if they differ.

The extension libraries only load into the OpenVINO version they were built against. Prepare a new model directory when DL Streamer moves to a new OpenVINO release.

bev_pool.xml must take the BEV-pool accumulator as an input (bev_accumulator), as g3dbevfusion_prepare_model.sh exports it; the element fails to start with a model exported without it.

Calibration JSON schema#

{
  "lidar": { "pad": 6 },
  "cameras": {
    "0": {
      "slot": 0,
      "name": "CAM_FRONT",
      "camera2lidar": [1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1, 0, 0, 0, 0, 1],
      "intrinsics": [1266.4, 0, 816.3, 0, 1266.4, 491.5, 0, 0, 1]
    }
  }
}

The example shows one camera with placeholder values; a fusion rig lists six, keyed "0" to "5" for pads 0-5.

  • camera2lidar is a 4x4 row-major matrix (16 values) and intrinsics a 3x3 row-major matrix (9 values).

  • lidar.pad and the keys of cameras are gvastreammux sink pad indices (mux.sink_<pad>).

  • slot is the camera’s position in the model’s batch, in the nuScenes order CAM_FRONT, CAM_FRONT_RIGHT, CAM_FRONT_LEFT, CAM_BACK, CAM_BACK_LEFT, CAM_BACK_RIGHT. Every slot 0-5 must appear exactly once.

  • camera2lidar takes camera coordinates (x right, y down, z forward) to the LiDAR frame.

  • Omit cameras (or leave it empty) for a LiDAR-only rig.

Per-frame table#

A calibration file can also carry a frames array, which describes the sequence frame by frame:

{
  "lidar": { "pad": 6 },
  "cameras": { "0": { "slot": 0, "name": "CAM_FRONT", "camera2lidar": [...], "intrinsics": [...] }, ... },
  "frames": [
    {
      "timestamp_us": 0,
      "lidar": { "points": 254613, "sweeps": [ { "points": 34720, "time_lag": 0.0 }, { "points": 21834, "time_lag": 0.049763 }, ... ] },
      "cameras": { "0": { "camera2lidar": [...] }, ... }
    },
    ...
  ]
}

Entry i describes the LiDAR buffer whose LidarMeta frame_id is i, which is the file index for g3dlidarparse. Every key in an entry is optional:

  • lidar.points is the buffer’s point count. A buffer with a different count, or a frame_id beyond the table, is an error. This catches a LiDAR stream that does not match the table, for example one that does not start at the table’s first frame.

  • lidar.sweeps lists the LiDAR turns the point cloud is made of, in point order, with each turn’s time lag in seconds behind the frame. BEVFusion reads the lag as a fifth point column; without sweeps it is 0 for every point.

  • cameras gives each camera’s camera2lidar at its own exposure. It must cover every declared camera. Without it, the static camera2lidar applies.

  • timestamp_us, the capture time, becomes the PTS of the batch and of its LiDAR buffer. Without it, the PTS is unchanged.

The per-frame camera2lidar matters because each nuScenes camera exposes up to 45 ms before the LiDAR timestamp, while the vehicle moves. A static camera2lidar misplaces the camera features by about 0.1 m typically, and up to 0.4 m. The g3d/calibration event downstream carries the static calibration only.

nuscenes_to_sequence.py writes calibration.json (six cameras) and calibration_lidar_only.json for each converted scene, both with the table. See the sample README for how it builds the frames.

Pipeline Examples#

Camera + LiDAR fusion on a converted nuScenes scene#

./g3dbevfusion.sh nuscenes_mini_seq_12Hz/scene-0103 bevfusion_ov_model_2026.5.0/g3dbevfusion.json fusion -1 out.mp4

which runs:

gst-launch-1.0 -e \
  multifilesrc location=scene/CAM_FRONT/%06d.jpg caps=image/jpeg,framerate=12/1 ! jpegparse ! \
    jpegdec idct-method=islow ! videoconvert ! video/x-raw,format=BGR ! queue ! mux.sink_0 \
  ... the other five cameras on mux.sink_1 .. mux.sink_5 ... \
  multifilesrc location=scene/LIDAR_TOP/%06d.bin caps=application/octet-stream ! \
    g3dlidarparse frame-rate=12 ! queue ! mux.sink_6 \
  gvastreammux name=mux output-mode=container sync-mode=first-pts max-wait-time=5000000000 ! \
  g3dbevfusion config=g3dbevfusion.json calibration=scene/calibration.json ! \
  g3drender forward-axis=y point-stride=8 width=2400 height=900 ! videoconvert ! video/x-raw,format=NV12 ! \
  vah264enc rate-control=cqp ! h264parse ! mp4mux ! filesink location=out.mp4

Every branch must use the same frame rate: the streams are paired by timestamp, and the timestamps come from the frame index. g3dbevfusion then replaces them with the recorded capture times from the calibration’s per-frame table. That is why the MP4 plays at the recorded pace, with frames 50 or 100 ms apart.

LiDAR-only#

gst-launch-1.0 -e \
  multifilesrc location=scene/LIDAR_TOP/%06d.bin caps=application/octet-stream ! \
    g3dlidarparse frame-rate=12 ! queue ! mux.sink_6 \
  gvastreammux name=mux output-mode=container sync-mode=first-pts ! \
  g3dbevfusion config=g3dbevfusion.json calibration=scene/calibration_lidar_only.json ! fakesink

Input / Output#

Capability

Sink

multistream/x-analytics-batch(meta:GstAnalyticsBatchMeta) — a gvastreammux container batch

Src

multistream/x-analytics-batch(meta:GstAnalyticsBatchMeta) — the same batch, with detections added

Inside the batch:

  • LiDAR: application/x-lidar from g3dlidarparse or g3dlidarsrc, with LidarMeta. Points are (x, y, z, intensity) float32. nuScenes intensities (0..255) are used as-is, as the model was trained on them. A point cloud may hold several LiDAR turns, already moved into the frame’s LiDAR coordinates; the calibration’s per-frame table gives their time lags.

  • Cameras: video/x-raw, format=BGR in system memory. Images are resized by 0.48 and the bottom-centre 704×256 is cropped, as in training, so they must be at least 1467×534 (nuScenes is 1600×900).

Metadata#

One GstAnalytics3DODMtd per detection is added to the LiDAR buffer’s GstAnalyticsRelationMeta:

  • x, y, z: box centre in the LiDAR frame (metres)

  • length, width, height: extents along the box’s own x, y and z axes (metres)

  • yaw: counter-clockwise rotation about +z from the LiDAR’s +x axis to the box’s length axis; pitch and roll are 0

  • class_id: index into the model’s classes: car, truck, construction_vehicle, bus, trailer, barrier, motorcycle, bicycle, pedestrian, traffic_cone

  • confidence: detection score

  • modality: GST_ANALYTICS_3D_SENSOR_LIDAR

The model also predicts a velocity per box. It is not attached, because GstAnalytics3DODMtd has no field for it.

Calibration propagated downstream (g3d/calibration event)#

Before the first buffer, the element pushes a sticky GST_EVENT_CUSTOM_DOWNSTREAM_STICKY event named g3d/calibration, in the same layout as g3dobjectfuser:

  • modality = "lidar", cameras = number of cameras

  • camera-<pad>: index, name, and the camera’s projection as tr_velo_to_cam = inverse(camera2lidar), r0_rect = identity, p2 = [K | 0]

  • class-names: array of strings, one per class_id

Limitations#

  • Six fixed cameras. The camera branch is exported for a batch of six images of 1600×900-class resolution.

  • LiDAR sweeps are accumulated upstream. The model is trained on a frame’s LiDAR turn plus the 9 before it, moved into the frame’s coordinates using the vehicle’s motion. The element does not accumulate turns itself. It takes the accumulated point cloud, and the time lag of each turn from the calibration’s per-frame table, as nuscenes_to_sequence.py writes them. A single turn with lag 0 also works, at lower recall.

  • One frame at a time. Inference is synchronous in the streaming thread. For fusion, the LiDAR and camera branches run in parallel inside the element. On an Intel Arc Pro B60, with ten accumulated sweeps, a frame takes about 90 ms for fusion and about 70 ms for LiDAR-only. Most of the LiDAR branch is the sparse encoder, at about 50 ms for 80,000 voxels.

  • One sparse-encoder context per process. The sparse-encoder extension keeps process-wide OpenCL state. Several g3dbevfusion instances in one process are serialised around it.

  • JPEG decoding matters. The camera branch is sensitive to small pixel differences. jpegdec and OpenCV decode the same JPEG slightly differently, and scores can move by a few hundredths as a result.

Debugging#

  • G3DBEVFUSION_DUMP_DIR=<dir> writes each frame’s decoded boxes as boxes_<frame>.npy: float32 [N, 11] holding x, y, z_bottom, dx, dy, dz, yaw, vx, vy, score, label.

  • With G3DBEVFUSION_DUMP_TENSORS=1, every stage’s input and output tensors are written too. That is about 90 MB per frame.

  • BEVFUSION_VERBOSE=1 prints the sparse-encoder extension’s timing and kernel selection.

Element Details (gst-inspect-1.0)#

Pad Templates:
  SINK template: 'sink'
    Availability: Always
    Capabilities:
      multistream/x-analytics-batch(meta:GstAnalyticsBatchMeta)

  SRC template: 'src'
    Availability: Always
    Capabilities:
      multistream/x-analytics-batch(meta:GstAnalyticsBatchMeta)

Element has no clocking capabilities.
Element has no URI handling capabilities.

Pads:
  SINK: 'sink'
    Pad Template: 'sink'
  SRC: 'src'
    Pad Template: 'src'

Element Properties:

  calibration         : JSON naming the lidar's and each camera's mux pad, with each camera's model slot, camera2lidar and intrinsics, and optionally a per-frame table (lidar sweeps, camera2lidar, capture time)
                        flags: readable, writable
                        String. Default: null
  config              : Runtime JSON: model_dir, extensions and gpu_custom_layers of the OpenVINO BEVFusion model
                        flags: readable, writable
                        String. Default: null
  device              : Device for the dense networks (camera encoder, geometry, detection head); the custom voxelize/bev_pool ops always run on the GPU
                        flags: readable, writable
                        String. Default: "GPU"
  name                : The name of the object
                        flags: readable, writable
                        String. Default: "g3dbevfusion0"
  parent              : The parent of the object
                        flags: readable, writable
                        Object of type "GstObject"
  qos                 : Handle Quality-of-Service events
                        flags: readable, writable
                        Boolean. Default: false
  score-threshold     : Minimum detection score to attach
                        flags: readable, writable
                        Float. Range:               0 -               1 Default:             0.3