# Performance Testing & Benchmarking Test your Order Accuracy pipeline performance on various hardware configurations. This guide covers everything from quick performance checks to comprehensive system capacity testing. ## Quick Start (5 minutes) **Goal**: Run a basic performance test to verify your system works correctly > [!IMPORTANT] > > **Inference Device:** The default device is `GPU`. To switch to `CPU`, you must do **both** steps below, otherwise the model will be exported for the wrong device: > > ::::{tab-set} > :::{tab-item} **Dine-In** > :sync: dine-in > > 1. Set this variable in your `.env` file: > > ```bash > TARGET_DEVICE=CPU # used by setup_models.sh and docker-compose > ``` > > 2. Re-export the model for the new device: > > ```bash > cd ../ovms-service && ./setup_models.sh --app dine-in > ``` > > `TARGET_DEVICE` is what `setup_models.sh` reads to export the model in the correct format. > > ::: > :::{tab-item} **Take-Away** > :sync: take-away > > 1. Set **both** variables in your `.env` file: > > ```bash > TARGET_DEVICE=CPU # used by setup_models.sh and docker-compose > OPENVINO_DEVICE=CPU # used by the Makefile benchmark targets > ``` > > 2. Re-export the model for the new device: > > ```bash > cd ../ovms-service && ./setup_models.sh --app take-away > ``` > > `TARGET_DEVICE` is what `setup_models.sh` reads to export the model in the correct format. `OPENVINO_DEVICE` is what the Makefile passes to the benchmark script. Both must match. > > ::: > :::: ### 1. Initialize Performance Tools ::::{tab-set} :::{tab-item} **Dine-In** :sync: dine-in ```bash # 1. Initialize git submodules (first time only) cd dine-in make update-submodules # 2. Start services make up ``` ::: :::{tab-item} **Take-Away** :sync: take-away ```bash # 1. Initialize git submodules (first time only) cd take-away make update-submodules # 2. Start services make up ``` ::: :::: ### 2. Run Quick Benchmark ::::{tab-set} :::{tab-item} **Dine-In** :sync: dine-in > [!NOTE] > The `images/` folder contains some sample images for testing. To use your own images, add them before testing: > > 1. Place plate images in `images/` (`.jpg`, `.jpeg`, or `.png`) > 2. Edit `configs/orders.json` — add entries with `image_id` matching your filenames > 3. Edit `configs/inventory.json` — define all possible menu items ```bash make benchmark ``` > [!NOTE] > `make benchmark` uses Docker profiles to start worker containers. Both the `dine-in` app and `dinein-worker` services use the **same Docker image** (built from the same Dockerfile). The worker is simply the same container running `worker.py` instead of the UI. **Variables:** | Variable | Default | Description | | ----------------------------- | ------- | ---------------------- | | `BENCHMARK_WORKERS` | 1 | Concurrent workers | | `BENCHMARK_DURATION` | 180 | Duration (seconds) | | `BENCHMARK_TARGET_LATENCY_MS` | 10000 | Latency threshold (ms) | | `TARGET_DEVICE` | GPU | Device: CPU, GPU | ::: :::{tab-item} **Take-Away** :sync: take-away > [!IMPORTANT] > Before running benchmarks, ensure a test video file is present at `storage/videos/test.mp4`. You can download a sample video using: > > ```bash > make download-sample-video > ``` > [!CAUTION] > **Order manifests:** The benchmark validates the orders detected in the video > against `config/orders.json`. Order IDs must match the order numbers shown in > the video, and each order's expected items must match what is actually visible. > A stale manifest reports correct detections as mismatches; an unparseable one > (for example a trailing comma) makes every order fail and the benchmark reports > zero transactions. ```bash # Default run make benchmark ``` ::: :::: ## Understanding Benchmark Types ::::::{tab-set} :::::{tab-item} **Dine-In Benchmarks** :sync: dine-in ::::{tab-set} :::{tab-item} **Single Request Benchmark** :sync: single ```bash make benchmark-single IMAGE_ID=MCD-1001 ``` Tests single image validation latency: - Image preprocessing time - VLM inference time - Semantic matching time - Total end-to-end latency ::: :::{tab-item} **Stream Density Benchmark** :sync: density ```bash make benchmark-stream-density # With overrides make benchmark-stream-density BENCHMARK_TARGET_LATENCY_MS=20000 BENCHMARK_INIT_DURATION=30 ``` > [!NOTE] > `make benchmark-stream-density` runs a Python script locally that sends concurrent HTTP requests to the running `dine-in` API. No separate worker containers are needed for this mode. Finds maximum concurrent requests the system can handle under latency constraints: - Target latency threshold (configurable) - Progressive load increase - Identifies performance ceiling ::: :::: ::::: :::::{tab-item} **Take-Away Benchmarks** :sync: take-away ::::{tab-set} :::{tab-item} **Single Video Benchmark** :sync: single ```bash make benchmark ``` Tests end-to-end latency for single order validation: - Video upload time - Frame extraction time - VLM inference latency - Validation time - Total processing time ::: :::{tab-item} **Fixed Workers Benchmark** Runs `benchmark_order_accuracy.py` with a fixed number of concurrent workers. ```bash # Default run make benchmark # Custom run make benchmark \ BENCHMARK_WORKERS=4 \ BENCHMARK_DURATION=300 \ BENCHMARK_INIT_DURATION=30 ``` Tests system with fixed number of concurrent workers: - Tests GPU/CPU performance for order validation - Measures end-to-end latency - Generates performance metrics - Outputs results to `results/` directory ::: :::{tab-item} **Stream Density Benchmark** :sync: density Finds the maximum number of concurrent workers the system can sustain under a target latency threshold. Runs `stream_density_latency_oa.py`. ```bash # Default run make benchmark-stream-density # Custom run make benchmark-stream-density \ BENCHMARK_TARGET_LATENCY_MS=10000 \ BENCHMARK_LATENCY_METRIC=avg \ BENCHMARK_INIT_DURATION=30 \ BENCHMARK_MIN_TRANSACTIONS=3 \ BENCHMARK_WORKER_INCREMENT=1 ``` Finds maximum sustainable worker count under latency constraints: - Maximum concurrent workers - Latency at each worker count - Point of degradation - Resource utilization at capacity ::: :::: ::::: :::::: ## Environment Variables Reference ::::{tab-set} :::{tab-item} **Dine-In Configuration** :sync: dine-in | Variable | Default | Description | | ----------------------------- | ----------------------- | ------------------------------------ | | `BENCHMARK_TARGET_LATENCY_MS` | `10000` | Target latency threshold (ms) | | `BENCHMARK_LATENCY_METRIC` | `avg` | `avg`, `p95`, or `max` | | `BENCHMARK_DENSITY_INCREMENT` | `1` | Concurrent images per iteration | | `BENCHMARK_INIT_DURATION` | `60` | Warmup time (seconds) | | `BENCHMARK_MIN_REQUESTS` | `3` | Min requests before measuring | | `BENCHMARK_REQUEST_TIMEOUT` | `300` | Individual request timeout (seconds) | | `BENCHMARK_API_ENDPOINT` | `http://localhost:8083` | API endpoint URL | | `RESULTS_DIR` | `./results` | Results output directory | ::: :::{tab-item} **Take-Away Configuration** :sync: take-away | Variable | Default | Description | | ----------------------------- | ------- | ------------------------------------------------------ | | `BENCHMARK_TARGET_LATENCY_MS` | `10000` | Target latency threshold (ms) | | `BENCHMARK_LATENCY_METRIC` | `avg` | Metric to evaluate: `avg` or `p95` | | `BENCHMARK_WORKER_INCREMENT` | `1` | Workers added per iteration | | `BENCHMARK_INIT_DURATION` | `10` | Warmup time per iteration (seconds) | | `BENCHMARK_MIN_TRANSACTIONS` | `1` | Min transactions before measuring latency | | `BENCHMARK_WORKERS` | `1` | Number of workers (fixed mode) | | `BENCHMARK_DURATION` | `200` | Test duration (seconds) | | `OOM_PROTECTION` | `1` | Set to `0` to disable OOM protection (not recommended) | ::: :::: ## Hardware Testing Commands ### GPU Performance Testing ::::{tab-set} :::{tab-item} **Dine-In** :sync: dine-in ```bash # Ensure GPU device is configured in .env # OPENVINO_DEVICE=GPU make benchmark ``` ::: :::{tab-item} **Take-Away** :sync: take-away ```bash # Configure GPU in .env # OPENVINO_DEVICE=GPU make benchmark-oa BENCHMARK_WORKERS=4 ``` ::: :::: ### Multi-Worker Stress Testing (Take-Away) ```bash # Test with 2 parallel workers make up-parallel WORKERS=2 make benchmark-oa BENCHMARK_WORKERS=2 # High stress test with 8 workers make up-parallel WORKERS=8 make benchmark-oa BENCHMARK_WORKERS=8 ``` ### Progressive Load Testing ```bash # Automatically find maximum sustainable workers make benchmark-stream-density \ BENCHMARK_TARGET_LATENCY_MS=10000 \ BENCHMARK_WORKER_INCREMENT=1 \ BENCHMARK_MAX_ITERATIONS=20 ``` ## Viewing Results and Metrics ::::{tab-set} :::{tab-item} **Dine-In Results** :sync: dine-in ```bash # View density benchmark results make benchmark-density-results # View raw results cat results/benchmark_results.json ls -la results/ ``` ::: :::{tab-item} **Take-Away Results** :sync: take-away Results are saved to the `results/` directory: ```text results/ ├── vlm_application_metrics_*.txt # VLM application metrics ├── vlm_performance_metrics_*.txt # VLM performance metrics └── consolidated_metrics.csv # Generated by make consolidate-metrics ``` ```bash # View VLM metrics make benchmark-oa-metrics # View benchmark results make benchmark-oa-results ``` ::: :::: ### Consolidate Metrics Consolidate metrics from multiple runs into a single CSV file: ```bash make consolidate-metrics ``` Generate plots from the consolidated metrics: ```bash make plot-metrics ``` ## Expected Performance ### Typical Latency Ranges | Operation | Dine-In | Take-Away | | ----------------------- | --------- | --------------- | | **Image Preprocessing** | 100-500ms | N/A | | **Frame Selection** | N/A | 200-500ms | | **VLM Inference** | 5-10s | 5-10s | | **Semantic Matching** | 50-200ms | 50-200ms | | **Total End-to-End** | 8-15s | 8-15s per order | ### Hardware Impact | Configuration | Typical Performance | | ------------------ | --------------------- | | **CPU Only** | 15-25s per validation | | **Intel iGPU** | 8-15s per validation | | **Intel Arc dGPU** | 5-10s per validation | | **NVIDIA RTX** | 4-8s per validation | ### Throughput Expectations | Mode | Expected Throughput | | ---------------------------------- | ------------------- | | **Dine-In Single** | 4-6 orders/minute | | **Take-Away Single** | 4-6 orders/minute | | **Take-Away Parallel (4 workers)** | 16-24 orders/minute | | **Take-Away Parallel (8 workers)** | 30-40 orders/minute | ## Optimization Tips ### GPU Utilization - Monitor GPU usage with `nvidia-smi -l 1` or `intel_gpu_top` - Target 70-90% GPU utilization for optimal throughput - If GPU is underutilized, increase worker count ### Memory Management - Monitor container memory with `docker stats` - VLM models require 8-16GB GPU memory - Reduce batch size if out-of-memory errors occur ### Network Optimization (Take-Away) - Use wired connections for RTSP streams - Ensure 1Gbps+ network bandwidth per camera - Consider local video storage for testing ### Latency Reduction - Use INT8 model quantization - Enable HTTP/2 for API connections - Pre-warm VLM model before benchmarking ## Troubleshooting Performance Issues ### Low FPS / High Latency - Check GPU driver installation - Verify OPENVINO_DEVICE setting in .env - Reduce image resolution or batch size - Check for thermal throttling ### VLM Timeout Errors - Increase API_TIMEOUT in .env - Check GPU memory availability - Consider using smaller model precision ### Memory Exhaustion - Reduce number of parallel workers - Lower batch size settings - Monitor with `docker stats` ### Inconsistent Results - Increase warmup duration (INIT_DURATION) - Increase minimum transactions (MIN_TRANSACTIONS) - Run multiple benchmark iterations