# Benchmarking

The POI system includes two benchmark targets that measure end-to-end
detection-to-alert latency. Both targets accept the same `DEVICE` parameter as
`make init` so the benchmark always runs with the correct model precision and
hardware profile.

## Prerequisites

Before running any benchmark, ensure the following setup steps have been
completed:

```bash
# 1. Initialize Git submodules (performance-tools, SceneScape, etc.)
make update-submodules

# 2. Download required AI models (face detection, re-identification)
make download-models

# 3. Generate configs and build images for your target device
make init                         # GPU (default)
make init DEVICE=all-cpu.env      # CPU variant
make init DEVICE=all-gpu.env      # GPU (explicit)
make init DEVICE=all-npu-cpu.env  # or NPU variant
```

**Sample data requirements:**

- **SceneScape video files** — sample video files must be present in the
  SceneScape sample data directory (e.g., `scenescape/sample_data/`). These
  provide the camera feed for the benchmark pipeline.
- **POI sample images** — at least `sample_data/poi_1.png` must exist in the
  POI directory. The benchmark enrolls this image as the target POI.

> **Note:** If models are missing or sample data is absent, the benchmark will
> fail during service startup or POI enrollment.

## `make benchmark` — Single-Scene Latency

Measures how quickly a matched POI triggers an alert with **one camera scene**.

```bash
# GPU (default)
make benchmark

# CPU — all pipelines on CPU, FP32
make benchmark DEVICE=all-cpu.env

# GPU — detect+reid on GPU, FP16
make benchmark DEVICE=all-gpu.env

# GPU detect / CPU reid, FP16
make benchmark DEVICE=all-gpu-cpu.env

# NPU detect / CPU reid, FP16-INT8
make benchmark DEVICE=all-npu-cpu.env

# NPU detect+reid, FP16-INT8
make benchmark DEVICE=all-npu.env
```

What it does:
1. Tears down any running stack and removes stale POI volumes.
2. Runs `make init DEVICE=<device>` to regenerate configs with the correct device and precision.
3. Starts all services (`make up`).
4. Enrolls a sample POI from `sample_data/poi_1.png`.
5. Waits for a detection-to-alert event (exits early on first match).
6. Deletes the benchmark POI and cleans up.

Tune timing with:

```bash
make benchmark DEVICE=all-gpu.env \
  BENCHMARK_TARGET_LATENCY_MS=500 \
  BENCHMARK_DURATION=180
```

## `make benchmark-stream-density` — Scaling Benchmark

Iteratively increases the number of camera scenes until the detection-to-alert
latency exceeds `BENCHMARK_TARGET_LATENCY_MS`, finding the maximum supported
stream density.

```bash
# GPU (default)
make benchmark-stream-density

# CPU
make benchmark-stream-density DEVICE=all-cpu.env

# GPU (explicit)
make benchmark-stream-density DEVICE=all-gpu.env

# GPU detect / CPU reid
make benchmark-stream-density DEVICE=all-gpu-cpu.env \
  BENCHMARK_SCENE_INCREMENT=2 \
  BENCHMARK_TARGET_LATENCY_MS=3000
```

The `DEVICE` setting is propagated to every `init.sh` call during scale-up
iterations — no manual re-init is required.

## Benchmark Parameters

| Parameter | Default | Description |
|---|---|---|
| `DEVICE` | `all-gpu.env` | Hardware profile — controls device, model precision, and pre-process backend |
| `BENCHMARK_TARGET_LATENCY_MS` | `30000` | Latency threshold in ms |
| `BENCHMARK_LATENCY_METRIC` | `avg` | Latency metric: `avg` or `max` |
| `BENCHMARK_DURATION` | `120` | Max wait (seconds) for single-run benchmark |
| `BENCHMARK_SCENE_INCREMENT` | `1` | Number of scenes added per iteration |
| `BENCHMARK_INIT_DURATION` | `120` | Warm-up seconds after each scale step |
| `BENCHMARK_STABILISE_DURATION` | `60` | Extra stabilisation wait after scale step |
| `BENCHMARK_MAX_ITERATIONS` | `50` | Safety cap on stream-density iterations |
| `RESULTS_PATH` | `./results` | Directory for CSV/JSON result files |

All parameters can also be set in the `benchmark{}` section of
`configs/zone_config.json` and are written to `docker/.env` during `make init`.

## Result Files

Results are written to `RESULTS_PATH` (default `./results/`) as both JSON and CSV:

```
results/
  poi_stream_density_20260526_135000.json
  poi_stream_density_20260526_135000.csv
  consolidated_metrics.csv                  # Generated by make consolidate-metrics
```

## Post-Processing Results

After running benchmarks, use the following targets to consolidate and visualize results:

### `make consolidate-metrics`

Merges results from multiple benchmark runs (across different devices, scene counts, etc.)
into a single CSV for comparison.

```bash
make consolidate-metrics
```

The consolidated CSV is written to `results/consolidated_metrics.csv`. This is useful when
comparing performance across hardware profiles (CPU vs GPU vs NPU) or across different
stream densities.

### `make plot-metrics`

Generates plots from benchmark results using `usage_graph_plot.py` from the
performance-tools submodule.

```bash
make plot-metrics
```

Plots are saved to `results/`. Requires Python with `matplotlib` installed in the
benchmark virtual environment (set up automatically by the benchmark targets).

## Quick Reference

```bash
# Run benchmarks
make benchmark                              # Single-scene latency (GPU, default)
make benchmark DEVICE=all-cpu.env           # Single-scene latency (CPU)
make benchmark DEVICE=all-gpu.env           # Single-scene latency (GPU, explicit)
make benchmark-stream-density               # Stream density scaling

# Post-process results
make consolidate-metrics                    # Merge runs into CSV
make plot-metrics                           # Generate plots

# View results
ls results/
```