Performance Testing & Benchmarking#

Test your Order Accuracy pipeline performance on various hardware configurations. This guide covers everything from quick performance checks to comprehensive system capacity testing.

Quick Start (5 minutes)#

Goal: Run a basic performance test to verify your system works correctly

Important

Inference Device: The default device is GPU. To switch to CPU, you must do both steps below, otherwise the model will be exported for the wrong device:

  1. Set this variable in your .env file:

TARGET_DEVICE=CPU      # used by setup_models.sh and docker-compose

  1. Re-export the model for the new device:

cd ../ovms-service && ./setup_models.sh --app dine-in

TARGET_DEVICE is what setup_models.sh reads to export the model in the correct format.

  1. Set both variables in your .env file:

TARGET_DEVICE=CPU      # used by setup_models.sh and docker-compose
OPENVINO_DEVICE=CPU    # used by the Makefile benchmark targets

  1. Re-export the model for the new device:

cd ../ovms-service && ./setup_models.sh --app take-away

TARGET_DEVICE is what setup_models.sh reads to export the model in the correct format. OPENVINO_DEVICE is what the Makefile passes to the benchmark script. Both must match.

1. Initialize Performance Tools#

# 1. Initialize git submodules (first time only)
cd dine-in
make update-submodules

# 2. Start services
make up

# 1. Initialize git submodules (first time only)
cd take-away
make update-submodules

# 2. Start services
make up

2. Run Quick Benchmark#

Note

The images/ folder contains some sample images for testing. To use your own images, add them before testing:

  1. Place plate images in images/ (.jpg, .jpeg, or .png)

  2. Edit configs/orders.json — add entries with image_id matching your filenames

  3. Edit configs/inventory.json — define all possible menu items

make benchmark

Note

make benchmark uses Docker profiles to start worker containers. Both the dine-in app and dinein-worker services use the same Docker image (built from the same Dockerfile). The worker is simply the same container running worker.py instead of the UI.

Variables:

Variable

Default

Description

BENCHMARK_WORKERS

1

Concurrent workers

BENCHMARK_DURATION

180

Duration (seconds)

BENCHMARK_TARGET_LATENCY_MS

25000

Latency threshold (ms)

TARGET_DEVICE

GPU

Device: CPU, GPU

Important

Before running benchmarks, ensure a test video file is present at storage/videos/test.mp4. You can download a sample video using:

make download-sample-video

Caution

Order manifests: The benchmark validates the orders detected in the video against config/orders.json. Order IDs must match the order numbers shown in the video, and each order’s expected items must match what is actually visible. A stale manifest reports correct detections as mismatches; an unparseable one (for example a trailing comma) makes every order fail and the benchmark reports zero transactions.

# Default run
make benchmark

Understanding Benchmark Types#

make benchmark-single IMAGE_ID=MCD-1001

Tests single image validation latency:

  • Image preprocessing time

  • VLM inference time

  • Semantic matching time

  • Total end-to-end latency

make benchmark-stream-density

# With overrides
make benchmark-stream-density BENCHMARK_TARGET_LATENCY_MS=20000 BENCHMARK_INIT_DURATION=30

Note

make benchmark-stream-density runs a Python script locally that sends concurrent HTTP requests to the running dine-in API. No separate worker containers are needed for this mode.

Finds maximum concurrent requests the system can handle under latency constraints:

  • Target latency threshold (configurable)

  • Progressive load increase

  • Identifies performance ceiling

make benchmark

Tests end-to-end latency for single order validation:

  • Video upload time

  • Frame extraction time

  • VLM inference latency

  • Validation time

  • Total processing time

Runs benchmark_order_accuracy.py with a fixed number of concurrent workers.

# Default run
make benchmark

# Custom run
make benchmark \
  BENCHMARK_WORKERS=4 \
  BENCHMARK_DURATION=300 \
  BENCHMARK_INIT_DURATION=30

Tests system with fixed number of concurrent workers:

  • Tests GPU/CPU performance for order validation

  • Measures end-to-end latency

  • Generates performance metrics

  • Outputs results to results/ directory

Finds the maximum number of concurrent workers the system can sustain under a target latency threshold. Runs stream_density_latency_oa.py.

# Default run
make benchmark-stream-density

# Custom run
make benchmark-stream-density \
  BENCHMARK_TARGET_LATENCY_MS=25000 \
  BENCHMARK_LATENCY_METRIC=avg \
  BENCHMARK_INIT_DURATION=30 \
  BENCHMARK_MIN_TRANSACTIONS=3 \
  BENCHMARK_WORKER_INCREMENT=1

Finds maximum sustainable worker count under latency constraints:

  • Maximum concurrent workers

  • Latency at each worker count

  • Point of degradation

  • Resource utilization at capacity

Environment Variables Reference#

Variable

Default

Description

BENCHMARK_TARGET_LATENCY_MS

25000

Target latency threshold (ms)

BENCHMARK_LATENCY_METRIC

avg

avg, p95, or max

BENCHMARK_DENSITY_INCREMENT

1

Concurrent images per iteration

BENCHMARK_INIT_DURATION

60

Warmup time (seconds)

BENCHMARK_MIN_REQUESTS

3

Min requests before measuring

BENCHMARK_REQUEST_TIMEOUT

300

Individual request timeout (seconds)

BENCHMARK_API_ENDPOINT

http://localhost:8083

API endpoint URL

RESULTS_DIR

./results

Results output directory

Variable

Default

Description

BENCHMARK_TARGET_LATENCY_MS

25000

Target latency threshold (ms)

BENCHMARK_LATENCY_METRIC

avg

Metric to evaluate: avg or p95

BENCHMARK_WORKER_INCREMENT

1

Workers added per iteration

BENCHMARK_INIT_DURATION

10

Warmup time per iteration (seconds)

BENCHMARK_MIN_TRANSACTIONS

1

Min transactions before measuring latency

BENCHMARK_WORKERS

1

Number of workers (fixed mode)

BENCHMARK_DURATION

200

Test duration (seconds)

OOM_PROTECTION

1

Set to 0 to disable OOM protection (not recommended)

Hardware Testing Commands#

GPU Performance Testing#

# Ensure GPU device is configured in .env
# OPENVINO_DEVICE=GPU
make benchmark

# Configure GPU in .env
# OPENVINO_DEVICE=GPU
make benchmark-oa BENCHMARK_WORKERS=4

Multi-Worker Stress Testing (Take-Away)#

# Test with 2 parallel workers
make up-parallel WORKERS=2
make benchmark-oa BENCHMARK_WORKERS=2

# High stress test with 8 workers
make up-parallel WORKERS=8
make benchmark-oa BENCHMARK_WORKERS=8

Progressive Load Testing#

# Automatically find maximum sustainable workers
make benchmark-stream-density \
  BENCHMARK_TARGET_LATENCY_MS=25000 \
  BENCHMARK_WORKER_INCREMENT=1 \
  BENCHMARK_MAX_ITERATIONS=20

Viewing Results and Metrics#

# View density benchmark results
make benchmark-density-results

# View raw results
cat results/benchmark_results.json
ls -la results/

Results are saved to the results/ directory:

results/
├── vlm_application_metrics_*.txt    # VLM application metrics
├── vlm_performance_metrics_*.txt    # VLM performance metrics
└── consolidated_metrics.csv         # Generated by make consolidate-metrics

# View VLM metrics
make benchmark-oa-metrics

# View benchmark results
make benchmark-oa-results

Consolidate Metrics#

Consolidate metrics from multiple runs into a single CSV file:

make consolidate-metrics

Generate plots from the consolidated metrics:

make plot-metrics

Expected Performance#

Typical Latency Ranges#

Operation

Dine-In

Take-Away

Image Preprocessing

100-500ms

N/A

Frame Selection

N/A

200-500ms

VLM Inference

5-10s

5-10s

Semantic Matching

50-200ms

50-200ms

Total End-to-End

8-15s

8-15s per order

Hardware Impact#

Configuration

Typical Performance

CPU Only

15-25s per validation

Intel iGPU

8-15s per validation

Intel Arc dGPU

5-10s per validation

NVIDIA RTX

4-8s per validation

Throughput Expectations#

Mode

Expected Throughput

Dine-In Single

4-6 orders/minute

Take-Away Single

4-6 orders/minute

Take-Away Parallel (4 workers)

16-24 orders/minute

Take-Away Parallel (8 workers)

30-40 orders/minute

Optimization Tips#

GPU Utilization#

  • Monitor GPU usage with nvidia-smi -l 1 or intel_gpu_top

  • Target 70-90% GPU utilization for optimal throughput

  • If GPU is underutilized, increase worker count

Memory Management#

  • Monitor container memory with docker stats

  • VLM models require 8-16GB GPU memory

  • Reduce batch size if out-of-memory errors occur

Network Optimization (Take-Away)#

  • Use wired connections for RTSP streams

  • Ensure 1Gbps+ network bandwidth per camera

  • Consider local video storage for testing

Latency Reduction#

  • Use INT8 model quantization

  • Enable HTTP/2 for API connections

  • Pre-warm VLM model before benchmarking

Troubleshooting Performance Issues#

Low FPS / High Latency#

  • Check GPU driver installation

  • Verify OPENVINO_DEVICE setting in .env

  • Reduce image resolution or batch size

  • Check for thermal throttling

VLM Timeout Errors#

  • Increase API_TIMEOUT in .env

  • Check GPU memory availability

  • Consider using smaller model precision

Memory Exhaustion#

  • Reduce number of parallel workers

  • Lower batch size settings

  • Monitor with docker stats

Inconsistent Results#

  • Increase warmup duration (INIT_DURATION)

  • Increase minimum transactions (MIN_TRANSACTIONS)

  • Run multiple benchmark iterations