Benchmarking Guide — Dine-In Order Accuracy#

This guide covers performance testing, stream density benchmarking, and metrics collection for the Dine-In Order Accuracy system.

Important

Inference Device: The default device is GPU. To switch to CPU, you must do both steps below, otherwise the model will be exported for the wrong device:

  1. Set this variable in your .env file:

    TARGET_DEVICE=CPU      # used by setup_models.sh and docker-compose

  2. Re-export the model for the new device:

    cd ../ovms-service && ./setup_models.sh --app dine-in

TARGET_DEVICE is what setup_models.sh reads to export the model in the correct format.

Prerequisites#

# 1. Initialize git submodules (first time only)
make update-submodules

# 2. Start services
make up

Note

The images/ folder contains some sample images for testing. To use your own images, add them before testing:

  1. Place plate images in images/ (.jpg, .jpeg, or .png)

  2. Edit configs/orders.json — add entries with image_id matching your filenames

  3. Edit configs/inventory.json — define all possible menu items

Benchmark Commands#

Single Image Test#

make benchmark-single IMAGE_ID=MCD-1001

Full Benchmark#

make benchmark

Note

make benchmark uses Docker profiles to start worker containers. Both the dine-in app and dinein-worker services use the same Docker image (built from the same Dockerfile). The worker is simply the same container running worker.py instead of the UI.

Variables:

Variable

Default

Description

BENCHMARK_WORKERS

1

Concurrent workers

BENCHMARK_DURATION

180

Duration (seconds)

BENCHMARK_TARGET_LATENCY_MS

25000

Latency threshold (ms)

TARGET_DEVICE

GPU

Device: CPU, GPU

Stream Density Benchmark#

Finds the maximum number of concurrent image validations within the latency target:

make benchmark-stream-density

# With overrides
make benchmark-stream-density BENCHMARK_TARGET_LATENCY_MS=20000 BENCHMARK_INIT_DURATION=30

Note

make benchmark-stream-density runs a Python script locally that sends concurrent HTTP requests to the running dine-in API. No separate worker containers are needed for this mode.

Metrics Processing#

# Consolidate metrics from multiple runs into a single CSV file
make consolidate-metrics

# Generate plots from the consolidated metrics
make plot-metrics