# Get Started Guide This article describes how to run Metro AI Suite Sensor Fusion for Traffic Management on Bare Metal systems. Before proceding, make sure to address the [prerequisites](./get-started-guide/prerequisites.md) and meet [system requirements](./get-started-guide/system-req.md). Metro AI Suite Sensor Fusion for Traffic Management application can support different pipeline using topology JSON files to describe the pipeline topology. The defined pipeline topology can be found at [Resources Summary](#resources-summary) ## Run Metro AI Suite Sensor Fusion for Traffic Management Application on Bare Metal systems There are two steps required for running the sensor fusion application: - Start AI Inference service, more details can be found at [Start Service](#start-service) - Run the application entry program, more details can be found at [Run Entry Program](#run-entry-program) Besides, you can test each component (without display) following the guides at [Advanced-User-Guide.md](./advanced-user-guide.md#entry-program) ### Resources Summary - Local File Pipeline for Media pipeline - Json File: localMediaPipeline.json > File location: `$PROJ_DIR/ai_inference/test/configs/kitti/1C1L/localMediaPipeline.json` - Pipeline Description: ``` input -> decode -> detection -> tracking -> output ``` - Local File Pipeline for Lidar pipeline - Json File: localLidarPipeline.json > File location: `$PROJ_DIR/ai_inference/test/configs/kitti/1C1L/localLidarPipeline.json` - Pipeline Description: ``` input -> lidar signal processing -> output ``` - Local File Pipeline for `Camera + Lidar(2C+1L)` Sensor fusion pipeline - Json File: localFusionPipeline.json > File location: `$PROJ_DIR/ai_inference/test/configs/kitti/2C1L/localFusionPipeline.json` - Pipeline Description: ``` | -> decode -> detector -> tracker -> | | input | -> decode -> detector -> tracker -> | -> LidarCam2CFusion -> fusion -> | -> output | -> lidar signal processing -> | | ``` - Local File Pipeline for `Camera + Lidar(4C+2L)` Sensor fusion pipeline - Json File: localFusionPipeline.json > File location: `$PROJ_DIR/ai_inference/test/configs/raddet/2C1L/localFusionPipeline.json` - Pipeline Description: ``` | -> decode -> detector -> tracker -> | | input | -> decode -> detector -> tracker -> | -> LidarCam2CFusion -> fusion -> | | -> lidar signal processing -> | | | -> decode -> detector -> tracker -> | | -> output input | -> decode -> detector -> tracker -> | -> LidarCam2CFusion -> fusion -> | | -> lidar signal processing -> | | ``` - Local File Pipeline for `Camera + Lidar(12C+2L)` Sensor fusion pipeline - Json File: localFusionPipeline.json `File location: ai_inference/test/configs/kitti/6C1L/localFusionPipeline.json` - Pipeline Description: ``` | -> decode -> detector -> tracker -> | | | -> decode -> detector -> tracker -> | | | -> decode -> detector -> tracker -> | | input | -> decode -> detector -> tracker -> | -> LidarCam6CFusion -> fusion -> | -> output | -> decode -> detector -> tracker -> | | | -> decode -> detector -> tracker -> | | | -> lidar signal processing -> | | ``` - Local File Pipeline for `Camera + Lidar(8C+4L)` Sensor fusion pipeline - Json File: localFusionPipeline.json `File location: ai_inference/test/configs/kitti/2C1L/localFusionPipeline.json` - Pipeline Description: ``` | -> decode -> detector -> tracker -> | | input | -> decode -> detector -> tracker -> | -> LidarCam2CFusion -> fusion -> | | -> lidar signal processing -> | | | -> decode -> detector -> tracker -> | | input | -> decode -> detector -> tracker -> | -> LidarCam2CFusion -> fusion -> | | -> lidar signal processing -> | | -> output | -> decode -> detector -> tracker -> | | input | -> decode -> detector -> tracker -> | -> LidarCam2CFusion -> fusion -> | | -> lidar signal processing -> | | | -> decode -> detector -> tracker -> | | input | -> decode -> detector -> tracker -> | -> LidarCam2CFusion -> fusion -> | | -> lidar signal processing -> | | ``` - Local File Pipeline for `Camera + Lidar(12C+4L)` Sensor fusion pipeline - Json File: localFusionPipeline.json `File location: ai_inference/test/configs/kitti/3C1L/localFusionPipeline.json` - Pipeline Description: ``` | -> decode -> detector -> tracker -> | | | -> decode -> detector -> tracker -> | | input | -> decode -> detector -> tracker -> | -> LidarCam3CFusion -> fusion -> | -> output | -> lidar signal processing -> | | ``` ### Start Service Open a terminal, run the following commands: ```bash cd $PROJ_DIR sudo bash -x run_service_bare.sh # Output logs: [2023-06-26 14:34:42.970] [DualSinks] [info] MaxConcurrentWorkload sets to 1 [2023-06-26 14:34:42.970] [DualSinks] [info] MaxPipelineLifeTime sets to 300s [2023-06-26 14:34:42.970] [DualSinks] [info] Pipeline Manager pool size sets to 1 [2023-06-26 14:34:42.970] [DualSinks] [trace] [HTTP]: uv loop inited [2023-06-26 14:34:42.970] [DualSinks] [trace] [HTTP]: Init completed [2023-06-26 14:34:42.971] [DualSinks] [trace] [HTTP]: http server at 0.0.0.0:50051 [2023-06-26 14:34:42.971] [DualSinks] [trace] [HTTP]: running starts [2023-06-26 14:34:42.971] [DualSinks] [info] Server set to listen on 0.0.0.0:50052 [2023-06-26 14:34:42.972] [DualSinks] [info] Server starts 1 listener. Listening starts [2023-06-26 14:34:42.972] [DualSinks] [trace] Connection handle with uid 0 created [2023-06-26 14:34:42.972] [DualSinks] [trace] Add connection with uid 0 into the conn pool ``` > NOTE-1 : workload (default as 4) can be configured in file: `$PROJ_DIR/ai_inference/source/low_latency_server/AiInference.config` ``` ... [Pipeline] maxConcurrentWorkload=4 ``` > NOTE-2 : to stop service, run the following commands: ```bash sudo pkill Hce ``` ### Run Entry Program #### Usage All executable files are located at: $PROJ_DIR/build/bin ##### entry program with display ``` Usage: CLSensorFusionDisplay [] [] [] [] [] -------------------------------------------------------------------------------- Environment requirement: unset http_proxy;unset https_proxy;unset HTTP_PROXY;unset HTTPS_PROXY ``` * **host**: use `127.0.0.1` to call from localhost. * **port**: configured as `50052`, can be changed by modifying file: `$PROJ_DIR/ai_inference/source/low_latency_server/AiInference.config` before starting the service. * **json_file**: AI pipeline topology file. * **total_stream_num**: to control the input streams. * **repeats**: to run tests multiple times, so that we can get more accurate performance. * **data_path**: multi-sensor binary files folder for input. * **display_type**: support for `media`, `lidar`, `media_lidar`, `media_fusion` currently. * `media`: only show image results in frontview. * `lidar`: only show lidar results in birdview. * `media_lidar`: show image results in frontview and lidar results in birdview separately. * `media_fusion`: show both for image results in frontview and fusion results in birdview. * **visualization_type**: visualization type of different pipelines, currently support `2C1L`, `4C2L`, `8C4L`, `12C2L`. * **save_flag**: whether to save display results into video. * **pipeline_repeats**: pipeline repeats number. * **cross_stream_num**: the stream number that run in a single pipeline. * **warmup_flag**: warm up flag before pipeline start. * **logo_flag**: whether to add intel logo in display. #### 2C+1L **The target platform is Intel® Core™ Ultra 7 265H.** > Note: Run with `root` if users want to get the GPU utilization profiling. > change /path-to-dataset to your data path. Please refer to [kitti360_guide.md](../../deployments/how_to_generate_kitti_format_dataset/kitti360_guide.md) for data preparation, or just use demo data in [kitti360](../../ai_inference/test/demo/kitti360/). - `media_fusion` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localFusionPipeline.json 1 1 /path-to-dataset media_fusion 2C1L ``` ![Display type: media_fusion](./_assets/2C1L-Display-type-media-fusion.png) - `media_lidar` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localFusionPipeline.json 1 1 /path-to-dataset media_lidar 2C1L ``` ![Display type: media_lidar](./_assets/2C1L-Display-type-media-lidar.png) - `media` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localMediaPipeline.json 1 1 /path-to-dataset media 2C1L ``` ![Display type: media](./_assets/2C1L-Display-type-media.png) - `lidar` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localLidarPipeline.json 1 1 /path-to-dataset lidar 2C1L ``` ![Display type: lidar](./_assets/2C1L-Display-type-lidar.png) #### 4C+2L **The target platform is Intel® Core™ Ultra 7 265H.** > Note: Run with `root` if users want to get the GPU utilization profiling. > change /path-to-dataset to your data path. Please refer to [kitti360_guide.md](../../deployments/how_to_generate_kitti_format_dataset/kitti360_guide.md) for data preparation, or just use demo data in [kitti360](../../ai_inference/test/demo/kitti360/). - `media_fusion` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localFusionPipeline.json 2 1 /path-to-dataset media_fusion 4C2L ``` ![Display type: media_fusion](./_assets/4C2L-Display-type-media-fusion.png) - `media_lidar` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localFusionPipeline.json 2 1 /path-to-dataset media_lidar 4C2L ``` ![Display type: media_lidar](./_assets/4C2L-Display-type-media-lidar.png) - `media` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localMediaPipeline.json 2 1 /path-to-dataset media 4C2L ``` ![Display type: media](./_assets/4C2L-Display-type-media.png) - `lidar` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localLidarPipeline.json 2 1 /path-to-dataset lidar 4C2L ``` ![Display type: lidar](./_assets/4C2L-Display-type-lidar.png) #### 12C+2L **Intel® Core™ i7-13700 and Intel® B580 Graphics.** > Note: Run with `root` if users want to get the GPU utilization profiling. > change /path-to-dataset to your data path. Please refer to [kitti360_guide.md](../../deployments/how_to_generate_kitti_format_dataset/kitti360_guide.md) for data preparation, or just use demo data in [kitti360](../../ai_inference/test/demo/kitti360/). - `media_fusion` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/6C1L/localFusionPipeline.json 2 1 /path-to-dataset media_fusion 12C2L ``` ![Display type: media_fusion](./_assets/12C2L-Display-type-media-fusion.png) - `media_lidar` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/6C1L/localFusionPipeline.json 2 1 /path-to-dataset media_lidar 12C2L ``` ![Display type: media_lidar](./_assets/12C2L-Display-type-media-lidar.png) - `media` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/6C1L/localMediaPipeline.json 2 1 /path-to-dataset media 12C2L ``` ![Display type: media](./_assets/12C2L-Display-type-media.png) - `lidar` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/6C1L/localLidarPipeline.json 2 1 /path-to-dataset lidar 12C2L ``` ![Display type: lidar](./_assets/12C2L-Display-type-lidar.png) #### 8C+4L **Intel® Core™ i7-13700 and Intel® B580 Graphics.** > Note: Run with `root` if users want to get the GPU utilization profiling. > change /path-to-dataset to your data path. Please refer to [kitti360_guide.md](../../deployments/how_to_generate_kitti_format_dataset/kitti360_guide.md) for data preparation, or just use demo data in [kitti360](../../ai_inference/test/demo/kitti360/). - `media_fusion` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localFusionPipeline.json 4 1 /path-to-dataset media_fusion 8C4L ``` ![Display type: media_fusion](./_assets/8C4L-Display-type-media-fusion.png) - `media_lidar` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localFusionPipeline.json 4 1 /path-to-dataset media_lidar 8C4L ``` ![Display type: media_lidar](./_assets/8C4L-Display-type-media-lidar.png) - `media` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localMediaPipeline.json 4 1 /path-to-dataset media 8C4L ``` ![Display type: media](./_assets/8C4L-Display-type-media.png) - `lidar` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/2C1L/localLidarPipeline.json 4 1 /path-to-dataset lidar 8C4L ``` ![Display type: lidar](./_assets/8C4L-Display-type-lidar.png) #### 12C+4L **Intel® Core™ i7-13700 and Intel® B580 Graphics.** > Note: Run with `root` if users want to get the GPU utilization profiling. > change /path-to-dataset to your data path. Please refer to [kitti360_guide.md](../../deployments/how_to_generate_kitti_format_dataset/kitti360_guide.md) for data preparation, or just use demo data in [kitti360](../../ai_inference/test/demo/kitti360/). - `media_fusion` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/3C1L/localFusionPipeline.json 4 1 /path-to-dataset media_fusion 12C4L ``` ![Display type: media_fusion](./_assets/12C4L-Display-type-media-fusion.png) - `media_lidar` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/3C1L/localFusionPipeline.json 4 1 /path-to-dataset media_lidar 12C4L ``` ![Display type: media_lidar](./_assets/12C4L-Display-type-media-lidar.png) - `media` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/3C1L/localMediaPipeline.json 4 1 /path-to-dataset media 12C4L ``` ![Display type: media](./_assets/12C4L-Display-type-media.png) - `lidar` display type open another terminal, run the following commands: ```bash # multi-sensor inputs test-case sudo -E ./build/bin/CLSensorFusionDisplay 127.0.0.1 50052 ai_inference/test/configs/kitti/3C1L/localLidarPipeline.json 4 1 /path-to-dataset lidar 12C4L ``` ![Display type: lidar](./_assets/12C4L-Display-type-lidar.png) ## Run Metro AI Suite Sensor Fusion for Traffic Management Application on Edge Microvisor Toolkit systems This section explains how to run Sensor Fusion for Traffic Management on Edge Microvisor Toolkit systems. For prerequisites and system requirements, please prepare a machine with the Edge Microvisor Toolkit system installed. **For Edge Microvisor Toolkit systems, Sensor Fusion for Traffic Management is only available in containerized format. To deploy and run the application on Edge Microvisor Toolkit, please follow the guidance bellow for pulling the docker image from DockerHub and running the containerized application.** ### Install X11 ```bash sudo -E tdnf install xorg-x11-server-Xorg xorg-x11-xinit xorg-x11-xinit-session xorg-x11-drv-libinput xorg-x11-apps xterm openbox libXfont2 freefont freetype gtk3 qemu-with-ui sudo dnf install python3 sudo -E python3 -m pip install PyXDG ``` ### Modify 20-modesetting.conf ```bash cd /usr/share/X11/xorg.conf.d/ sudo nano 20-modesetting.conf ## Add the following configuration into 20-modesetting.conf Section "Device" Identifier "Intel_Graphics" Driver "modesetting" Option "SWcursor" "true" Option "AccelMethod" "glamor" Option "DRI" "3" EndSection ``` ### X11 setting ```bash export XDG_RUNTIME_DIR=/tmp sudo -E bash -c 'xinit /usr/bin/openbox-session &' export DISPLAY=:0 xhost + xhost +local:docker ``` ### Pull docker image You can pull latest tfcc docker image through [intel/tfcc - Docker Image](https://hub.docker.com/r/intel/tfcc/). For example: ```bash docker pull intel/tfcc:latest ``` ### Run TFCC docker image on Edge Microvisor Toolkit systems For Edge Microvisor Toolkit systems, Sensor Fusion for Traffic Management is only available in containerized format. To deploy and run the application on Edge Microvisor Toolkit, please pulling the docker image from DockerHub and follow the guidance in the [run docker image](./advanced-user-guide.md#run-docker-image) section and [Running inside docker](./advanced-user-guide.md#running-inside-docker) section of [Advanced-User-Guide.md](./advanced-user-guide.md). ## Code Reference Some of the code is referenced from the following projects: - [IGT GPU Tools](https://gitlab.freedesktop.org/drm/igt-gpu-tools) (MIT License) - [Intel DL Streamer](https://github.com/dlstreamer/dlstreamer) (MIT License) - [Open Model Zoo](https://github.com/openvinotoolkit/open_model_zoo) (Apache-2.0 License) Current Version: 3.0 - Support 2C+1L/4C+2L pipeline - Support 8C+4L/12C+2L pipeline - Support Pointpillar model - Updated OpenVINO to 2025.3 - Updated oneAPI to 2025.3.0 :::{toctree} :hidden: Prerequisites System Requirements :::