Performance Testing & Benchmarking#
Test your Loss Prevention and Automated Self-Checkout pipeline performance on various hardware configurations. This guide covers everything from quick performance checks to comprehensive system capacity testing.
Quick Start (5 minutes)#
Goal: Run a basic performance test to verify your system works correctly
1. Initialize Performance Tools#
make update-submodules
2. Run Quick Benchmark#
make benchmark-quickstart
What this does:
Benchmarks the six workloads in the default camera configuration using the default mixed CPU/GPU/NPU workload mapping
Runs headless (no display needed)
Uses registry images by default (
REGISTRY=falsebuilds the benchmark image locally)Generates consolidated metrics in
benchmark/metrics.csv
Expected results: You’ll see FPS metrics and resource utilization for each workload.
Understanding Performance Testing Types#
Basic Performance Testing#
Default Benchmark Command:
make benchmark
Configuration:
Single pipeline instance (
PIPELINE_COUNT=1)Mixed CPU/GPU/NPU processing (
WORKLOAD_DIST=workload_to_pipeline.json)Standard camera setup (
CAMERA_STREAM=camera_to_workload.json)No visual output (
RENDER_MODE=0)
Environment Variables Reference#
Category |
Variable |
Description |
Common Values |
|---|---|---|---|
Display |
|
Show/hide visual output |
|
Performance |
|
Number of parallel pipeline instances |
|
Hardware |
|
Target processing hardware |
|
Camera Setup |
|
Camera configuration |
|
Build |
|
Use pre-built vs local images |
|
Workload Configuration Options#
Camera Stream Configurations#
Full Workload Testing (camera_to_workload.json):
Camera |
Workload |
|---|---|
cam1 |
items_in_basket |
cam2 |
hidden_items |
cam3 |
fake_scan_detection |
cam4 |
multi_product_identification |
cam5 |
product_switching |
cam6 |
sweet_heartening |
Hardware Distribution Options#
Configuration |
File |
Best For |
|---|---|---|
CPU Only |
|
Testing, development environments |
GPU Only |
|
Production, high performance |
Mixed GPU/NPU |
|
Assign selected workloads to GPU or NPU |
Heterogeneous |
|
Maximum performance across all hardware |
Default Mixed |
|
CPU/GPU/NPU distribution used by default |
Advanced Performance Testing (15-30 minutes)#
GPU Performance Testing#
make benchmark WORKLOAD_DIST=workload_to_pipeline_gpu.json CAMERA_STREAM=camera_to_workload.json
Multi-Pipeline Stress Testing#
# Test with 2 parallel pipelines
make PIPELINE_COUNT=2 benchmark
# High stress test with 4 pipelines
make PIPELINE_COUNT=4 benchmark
Custom Hardware Configuration#
# Test heterogeneous workload distribution
make benchmark WORKLOAD_DIST=workload_to_pipeline_hetero.json CAMERA_STREAM=camera_to_workload.json REGISTRY=false
Automated Self-Checkout Performance#
# Object detection workload
CAMERA_STREAM=camera_to_workload_asc_object_detection.json WORKLOAD_DIST=workload_to_pipeline_asc_object_detection_gpu.json make benchmark
# Age verification workload
CAMERA_STREAM=camera_to_workload_asc_age_verification.json WORKLOAD_DIST=workload_to_pipeline_asc_age_verification_gpu.json make benchmark
Viewing Results#
Generate Consolidated Metrics#
make consolidate-metrics
Output: benchmark/metrics.csv containing:
FPS (frames per second) for each pipeline
CPU/GPU/NPU utilization percentages
Memory usage statistics
Power consumption data
Latency measurements
View Results#
cat benchmark/metrics.csv
Stream Density Testing (30+ minutes)#
Goal: Find the maximum number of parallel pipelines your system can handle while maintaining target performance.
Basic Stream Density Test#
make benchmark-stream-density
Default behavior:
Requires every stream to reach at least 95% of its effective target FPS; by default targets come from each camera’s
targetFps, thenfps, with a fallback of15.Uses OOM protection to prevent system crashes
Reports maximum sustainable pipeline count
Custom Target FPS#
Set the target FPS for each camera using the targetFps field in the selected camera_to_workload_*.json file. Choose the value based on workload criticality and the source FPS.
If targetFps is not defined for a camera (or is not a positive value), the benchmark uses that camera’s fps field instead. If neither field provides a positive value, it falls back to the default target FPS of 15.
To override the camera configuration, set the TARGET_FPS environment variable. When supplied, TARGET_FPS applies the same target to every stream and takes precedence over both targetFps and fps.
In summary, the effective target FPS for each camera is resolved in this order:
TARGET_FPSenvironment variable, when explicitly supplied (overrides the camera configuration for every stream).The camera’s
targetFpsincamera_to_workload_*.json.The camera’s
fps, used whentargetFpsis missing or not a positive value.The default target FPS,
15, when neither camera field provides a positive value.
The benchmark passes when each stream reaches at least 95% of its effective target. The resolved target FPS and its source are recorded in stream_density.log.
CLI examples:
# Test for different performance thresholds
make TARGET_FPS=13.5 benchmark-stream-density
# With explicit camera and workload configurations
make CAMERA_STREAM=camera_to_workload.json WORKLOAD_DIST=workload_to_pipeline_gpu.json TARGET_FPS=13.5 benchmark-stream-density
Measurement interval and warmup period#
MEASUREMENT_WINDOW_SECONDS controls how long FPS samples are collected for each density step. Its default is 100 seconds, and values below 100 are rejected because the benchmark requires at least 100 samples per pipeline log. Each step adds a lane after one passing measurement interval, the final count is confirmed by run acceptance criterion (two consecutive passes), and a lane is only given up after two failing measurement interval in a row. Each measurement interval is evaluated independently; measurement interval are not combined into one longer measurement interval.
INIT_DURATION sets the warmup period, defaulting to 10 seconds, before measurement starts. A longer warmup period lets the pipelines stabilize more, and a longer measurement interval can reduce short-term measurement noise, but both increase the total benchmark duration. The resolved warmup period is also printed in the stream-density result summary.
make benchmark-stream-density \
INIT_DURATION=120 \
MEASUREMENT_WINDOW_SECONDS=120
Stream Density Environment Variables#
Variable |
Description |
Values |
|---|---|---|
|
Override the target FPS for all streams; unset uses camera configuration |
Unset by default, |
|
Duration FPS samples are collected per density step (minimum 100 seconds) |
|
|
warmup period before measurement starts |
|
|
Prevent out-of-memory crashes |
|
|
Fraction of each target FPS required to pass |
|
|
Pin all services to one core type (stream density only) |
|
|
Core type used when |
|
|
Number of round-robin model instances (stream density only) |
|
Warning: Setting
OOM_PROTECTION=0may crash your system requiring a hard reboot.
Core Pinning#
benchmark-stream-density can pin all services to a single core type (P-cores, E-cores or LP E-cores) on hybrid Intel CPUs. Core pinning is disabled by default and applies only to benchmark-stream-density.
# Pin to E-cores (default core type)
make benchmark-stream-density CORE_PINNING=true
# Override the round-robin model instance count (default: 128)
make benchmark-stream-density ROUND_ROBIN_COUNT=64
# Pin to P-cores or LP E-cores
make benchmark-stream-density CORE_PINNING=true CORE_TYPE=PCORE
make benchmark-stream-density CORE_PINNING=true CORE_TYPE=LPECORE
When CORE_PINNING=true:
src/get-core-types.sh <CORE_TYPE>returns the comma-separated core IDs for the selected core type. It reads/etc/core-typeswhen present; otherwise it downloadsobtain_cores.shat a pinned commit, verifies its SHA-256 checksum and runs it.The result is exported as
CPUSET, andsrc/docker-compose-core_pinning.ymlis applied as an override on top ofsrc/docker-compose.yml. The override setscpusetfor every service.ROUND_ROBIN_COUNTis configurable withmake benchmark-stream-density ROUND_ROBIN_COUNT=<count>and defaults to128.The selected core type, CPU set, and
ROUND_ROBIN_COUNTare written tocore_pinning.txtin the run’s results directory.
The benchmark stops with an error if CORE_TYPE is invalid, the selected core type is not present on the system, or CORE_PINNING is not true/false.
Note: Install
cpuid(sudo apt install cpuid) for the most reliable core-type detection; without it, detection falls back to sysfs andlscpu. Detection also requirescurland internet access unless/etc/core-typesexists.
To check the detected cores without running the benchmark:
src/get-core-types.sh # all core types, e.g. PCORE=0,1,2,3
src/get-core-types.sh ECORE # E-core IDs only, e.g. 4,5,6,7
src/get-core-types.sh --reinstall PCORE # ignore /etc/core-types and detect again
Note: When adding a service to
src/docker-compose.yml, add a matchingcpusetentry tosrc/docker-compose-core_pinning.ymlso it is pinned as well.
Note: stream-density caches camera-target mappings using (config_path, file_mtime) and auto-refreshes when the camera config changes; unit tests for this behavior are in the performance-tools repository at benchmark-scripts/stream_density_test.py.
Expected Output#
Stream density completed: use case density <lanes> lanes, stream density <streams> streams. For the detailed report, see benchmark/stream_density.log.
Visualization & Analysis#
Generate Performance Graphs#
make plot-metrics
Output: benchmark/plot_metrics.png showing:
CPU Usage Over Time
NPU Utilization Over Time
GPU Usage for each GPU device found
Useful Maintenance Commands#
make validate-all-configs # Validate configuration files
make clean-images # Remove dangling Docker images
make clean-containers # Remove stopped containers
make clean-all # Remove all unused Docker resources
Custom Configuration (Advanced)#
Creating Custom Workloads#
The application is highly configurable via JSON files in the configs/ directory:
Camera Configuration (camera_to_workload.json)#
{
"lane_config": {
"cameras": [
{
"camera_id": "cam1",
"fileSrc": "sample-media/video1.mp4",
"targetFps": 12.5,
"workloads": ["items_in_basket", "multi_product_identification"],
"region_of_interest": {"x": 100, "y": 100, "x2": 800, "y2": 600}
}
]
}
}
Pipeline Configuration (workload_to_pipeline.json)#
{
"workload_pipeline_map": {
"items_in_basket": [
{"type": "gvadetect", "model": "yolo11n", "precision": "INT8", "device": "CPU"},
{"type": "gvaclassify", "model": "efficientnet-v2-b0", "precision": "INT8", "device": "CPU"}
]
}
}
To Add Custom Workloads:#
Edit
configs/camera_to_workload.jsonto add your camera and assign workloadsEdit
configs/workload_to_pipeline.jsonto define the pipeline for your workloadPlace your video files in
performance-tools/sample-media/and update thefileSrcpathRun
make validate-all-configsto verify your configurationRe-run the pipeline:
make benchmark
Detailed Hardware Distribution (Reference)#
Heterogeneous Configuration Breakdown#
The workload_to_pipeline_hetero.json distributes workloads across multiple processing units:
Workload |
Object Detection |
Classification |
Inference |
|---|---|---|---|
items_in_basket |
CPU |
CPU |
- |
hidden_items |
GPU |
GPU |
- |
fake_scan_detection |
GPU |
GPU |
- |
multi_product_identification |
NPU |
NPU |
- |
product_switching |
CPU |
CPU |
- |
sweet_heartening |
NPU |
- |
NPU |
Mixed Configuration Details#
The workload_to_pipeline.json balances workloads across available hardware:
CPU: items_in_basket, multi_product_identification, sweet_heartening
GPU: product_switching, hidden_items
NPU: fake_scan_detection
Project Structure (Reference)#
configs/— Configuration files (camera/workload mapping)docker/— Dockerfiles for containersdownload-scripts/— Model and video download scriptssrc/— Pipeline runner scriptsbenchmark/— Benchmark logs, consolidated metrics, and plots (defaultRESULTS_DIR)Makefile— Build automation commands
Appendix: System Configuration#
Proxy Configuration (If Required)#
If your organization requires proxy settings for internet access:
Shell Session Proxy#
export http_proxy=http://<proxy-host>:<port>
export https_proxy=http://<proxy-host>:<port>
export HTTP_PROXY=http://<proxy-host>:<port>
export HTTPS_PROXY=http://<proxy-host>:<port>
export NO_PROXY=localhost,127.0.0.1,::1
System-Wide Proxy (/etc/environment)#
sudo nano /etc/environment
# Add the following lines:
http_proxy=http://<proxy-host>:<port>
https_proxy=http://<proxy-host>:<port>
HTTP_PROXY=http://<proxy-host>:<port>
HTTPS_PROXY=http://<proxy-host>:<port>
NO_PROXY=localhost,127.0.0.1,::1
Docker Proxy Configuration#
Create /etc/systemd/system/docker.service.d/http-proxy.conf:
sudo mkdir -p /etc/systemd/system/docker.service.d
sudo nano /etc/systemd/system/docker.service.d/http-proxy.conf
[Service]
Environment="http_proxy=http://<proxy-host>:<port>"
Environment="https_proxy=http://<proxy-host>:<port>"
Environment="no_proxy=localhost,127.0.0.1,::1"
# Reload and restart
sudo systemctl daemon-reload
sudo systemctl restart docker