Troubleshooting#

Common issues#

  • Startup fails with “model name must be provided”: Set EMBEDDING_MODEL_NAME before launching Docker (required for both SDK and API modes).

  • Object detection disabled unexpectedly: Check logs for YOLOX download failures. Ensure the YOLOX_MODELS_VOLUME_NAME volume exists and the host has outbound network access during first run.

  • API mode returns 502: Verify the multimodal embedding service is healthy at MULTIMODAL_EMBEDDING_ENDPOINT (see docker compose -f docker/compose-with-embedding.yaml ps).

  • Uploads rejected: Files larger than 500 MB are not accepted by the FastAPI upload endpoint. Stage the video directly in MinIO and use /media/process instead.

  • GPU acceleration inactive: Confirm /dev/dri/* is mapped into the container, set the relevant device variable (MM_DATAPREP_EMBEDDING_DEVICE or MM_DATAPREP_DETECTION_DEVICE) to GPU, and keep MM_DATAPREP_USE_OPENVINO=true.

  • NPU acceleration inactive: Confirm /dev/accel/accel0 is available on the host and mapped into the container, set the relevant device variable (MM_DATAPREP_EMBEDDING_DEVICE or MM_DATAPREP_DETECTION_DEVICE) to NPU, and keep MM_DATAPREP_USE_OPENVINO=true. Verify the selected model supports NPU inference via the OpenVINO Supported Models page.

  • First NPU run is slow (one-time model compilation): The first time a model runs on NPU, OpenVINO compiles it to an NPU-specific blob, which takes noticeably longer than CPU/GPU startup. This is expected and happens once per model/configuration. The compiled blob is cached on the OV_MODELS_DIR mount (default /app/ov_models), so subsequent runs reuse it and start quickly — persist this volume to retain the cache across container restarts.

4K/8K frames overflow the shared-memory block (worker timeout)#

Symptom. Ingesting a high-resolution (4K/8K) video stalls and the DataPrep worker is killed and rebooted by Gunicorn. The pipeline stage workers log that their queues never fill, then the worker aborts with signal 134 (SIGABRT) and leaks shared-memory objects:

WARNING: | detection_worker | [DETECTION QUEUE EMPTY] WAITING...
WARNING: | store_worker      | [STORE_WORKER] Queue empty, waiting...
WARNING: | embed_worker      | [EMBED_WORKER] Queue empty, waiting...
WARNING: | process_result_worker | [RESULT WORKER] Queue empty, waiting...
[CRITICAL] WORKER TIMEOUT (pid:8)
[ERROR] Worker (pid:8) was sent code 134!
UserWarning: resource_tracker: There appear to be 1024 leaked shared_memory objects to clean up at shutdown

Why it happens. In SDK mode the decoder transports each frame through a pre-allocated pool of fixed-size shared-memory blocks (SharedMemoryPool in src/core/embedding/decoder.py). Every decoded frame is written into one block as a raw RGB buffer of exactly width × height × 3 bytes. The block size is controlled by SDK_VIDEO_SHM_BLOCK_SIZE, which defaults to 6220800 = 1920 × 1080 × 3 (1080p). When a frame is larger than the block, the write into the too-small buffer fails inside the decode worker; the frame is never enqueued, so every downstream stage (detection → embed → store → result) sits on an empty queue and eventually the whole worker hits the Gunicorn timeout and is force-killed, orphaning the shared-memory blocks it had acquired.

Fix. Set SDK_VIDEO_SHM_BLOCK_SIZE to at least width × height × 3 for your highest-resolution source before sourcing the setup script (or bring the stack down and back up so the new value is applied):

Source resolution

Pixels (W × H)

Minimum SDK_VIDEO_SHM_BLOCK_SIZE (W × H × 3)

1080p (default)

1920 × 1080

6220800

4K UHD

3840 × 2160

24883200

DCI 4K

4096 × 2160

26542080

8K UHD

7680 × 4320

99532800

 
# Example: enable 4K ingestion (3840 x 2160 x 3 = 24883200 bytes per block)
export SDK_VIDEO_SHM_BLOCK_SIZE=24883200
source ./setup.sh          # or: source ./setup.sh --down && source ./setup.sh

Also budget the total shared memory. The pool pre-allocates SDK_VIDEO_SHM_MAX_BLOCKS × SDK_VIDEO_SHM_BLOCK_SIZE bytes in the host /dev/shm (the container runs with ipc: host). With the default 512 blocks, 4K needs ≈ 12.7 GB and 8K needs ≈ 51 GB of /dev/shm. If the host cannot spare that much, lower SDK_VIDEO_SHM_MAX_BLOCKS to keep the product within your available /dev/shm (check with df -h /dev/shm), for example:

export SDK_VIDEO_SHM_BLOCK_SIZE=24883200   # 4K frame size
export SDK_VIDEO_SHM_MAX_BLOCKS=128        # 128 x 24883200 ≈ 3.2 GB of /dev/shm

Tip: Pick the block size from the largest resolution you will ingest. A larger-than-needed block size is safe (it only wastes memory); a smaller one triggers the failure above. If you mix resolutions, size for the largest.