Benchmarking#

The POI system includes two benchmark targets that measure end-to-end detection-to-alert latency. Both targets accept the same DEVICE parameter as make init so the benchmark always runs with the correct model precision and hardware profile.

Prerequisites#

Before running any benchmark, ensure the following setup steps have been completed:

# 1. Initialize Git submodules (performance-tools, SceneScape, etc.)
make update-submodules

# 2. Download required AI models (face detection, re-identification)
make download-models

# 3. Generate configs and build images for your target device
make init                         # GPU (default)
make init DEVICE=all-cpu.env      # CPU variant
make init DEVICE=all-gpu.env      # GPU (explicit)
make init DEVICE=all-npu-cpu.env  # or NPU variant

Sample data requirements:

  • SceneScape video files — sample video files must be present in the SceneScape sample data directory (e.g., scenescape/sample_data/). These provide the camera feed for the benchmark pipeline.

  • POI sample images — at least sample_data/poi_1.png must exist in the POI directory. The benchmark enrolls this image as the target POI.

Note: If models are missing or sample data is absent, the benchmark will fail during service startup or POI enrollment.

make benchmark — Single-Scene Latency#

Measures how quickly a matched POI triggers an alert with one camera scene.

# GPU (default)
make benchmark

# CPU — all pipelines on CPU, FP32
make benchmark DEVICE=all-cpu.env

# GPU — detect+reid on GPU, FP16
make benchmark DEVICE=all-gpu.env

# GPU detect / CPU reid, FP16
make benchmark DEVICE=all-gpu-cpu.env

# NPU detect / CPU reid, FP16-INT8
make benchmark DEVICE=all-npu-cpu.env

# NPU detect+reid, FP16-INT8
make benchmark DEVICE=all-npu.env

What it does:

  1. Tears down any running stack and removes stale POI volumes.

  2. Runs make init DEVICE=<device> to regenerate configs with the correct device and precision.

  3. Starts all services (make up).

  4. Enrolls a sample POI from sample_data/poi_1.png.

  5. Waits for a detection-to-alert event (exits early on first match).

  6. Deletes the benchmark POI and cleans up.

Tune timing with:

make benchmark DEVICE=all-gpu.env \
  BENCHMARK_TARGET_LATENCY_MS=500 \
  BENCHMARK_DURATION=180

make benchmark-stream-density — Scaling Benchmark#

Iteratively increases the number of camera scenes until the detection-to-alert latency exceeds BENCHMARK_TARGET_LATENCY_MS, finding the maximum supported stream density.

# GPU (default)
make benchmark-stream-density

# CPU
make benchmark-stream-density DEVICE=all-cpu.env

# GPU (explicit)
make benchmark-stream-density DEVICE=all-gpu.env

# GPU detect / CPU reid
make benchmark-stream-density DEVICE=all-gpu-cpu.env \
  BENCHMARK_SCENE_INCREMENT=2 \
  BENCHMARK_TARGET_LATENCY_MS=3000

The DEVICE setting is propagated to every init.sh call during scale-up iterations — no manual re-init is required.

Benchmark Parameters#

Parameter

Default

Description

DEVICE

all-gpu.env

Hardware profile — controls device, model precision, and pre-process backend

BENCHMARK_TARGET_LATENCY_MS

30000

Latency threshold in ms

BENCHMARK_LATENCY_METRIC

avg

Latency metric: avg or max

BENCHMARK_DURATION

120

Max wait (seconds) for single-run benchmark

BENCHMARK_SCENE_INCREMENT

1

Number of scenes added per iteration

BENCHMARK_INIT_DURATION

120

Warm-up seconds after each scale step

BENCHMARK_STABILISE_DURATION

60

Extra stabilisation wait after scale step

BENCHMARK_MAX_ITERATIONS

50

Safety cap on stream-density iterations

RESULTS_PATH

./results

Directory for CSV/JSON result files

All parameters can also be set in the benchmark{} section of configs/zone_config.json and are written to docker/.env during make init.

Result Files#

Results are written to RESULTS_PATH (default ./results/) as both JSON and CSV:

results/
  poi_stream_density_20260526_135000.json
  poi_stream_density_20260526_135000.csv
  consolidated_metrics.csv                  # Generated by make consolidate-metrics

Post-Processing Results#

After running benchmarks, use the following targets to consolidate and visualize results:

make consolidate-metrics#

Merges results from multiple benchmark runs (across different devices, scene counts, etc.) into a single CSV for comparison.

make consolidate-metrics

The consolidated CSV is written to results/consolidated_metrics.csv. This is useful when comparing performance across hardware profiles (CPU vs GPU vs NPU) or across different stream densities.

make plot-metrics#

Generates plots from benchmark results using usage_graph_plot.py from the performance-tools submodule.

make plot-metrics

Plots are saved to results/. Requires Python with matplotlib installed in the benchmark virtual environment (set up automatically by the benchmark targets).

Quick Reference#

# Run benchmarks
make benchmark                              # Single-scene latency (GPU, default)
make benchmark DEVICE=all-cpu.env           # Single-scene latency (CPU)
make benchmark DEVICE=all-gpu.env           # Single-scene latency (GPU, explicit)
make benchmark-stream-density               # Stream density scaling

# Post-process results
make consolidate-metrics                    # Merge runs into CSV
make plot-metrics                           # Generate plots

# View results
ls results/