Get Started - Deploy Surgical Instrument#

This is a deployment guide for the three-service Docker stack:

  • surgical-backend (Flask 3 + Ultralytics + OpenVINO bootstrap)

  • surgical-pipeline (GStreamer + DL Streamer runtime)

  • surgical-ui (nginx + React SPA)

Prerequisites#

Before you start, refer to System Requirements to confirm your setup compatibility.

If you work behind a proxy, see how to configure the app for a proxy.

The application does not ship with the medical dataset, trained model binaries, or demo videos. You will need to collect these resources separately, as described in the following steps, and store them locally. This keeps the demo reproducible without any dependency on a specific machine.

1. Clone the repo and download the dataset#

Open PowerShell and run:

git clone --filter=blob:none --sparse https://github.com/open-edge-platform/edge-ai-suites.git `
; cd edge-ai-suites `
; git sparse-checkout set health-and-life-sciences-ai-suite/Surgical_Instrument `
; cd health-and-life-sciences-ai-suite/Surgical_Instrument

Download the dataset and place it in the following local folder for datasets (create it if it does not exist), for example: Surgical_Instrument/datasets/ColonDB/raw/.

Archive support:

  • .zip, .tar, .tar.gz, .tgz can be consumed directly.

  • .rar must be extracted locally.

The bootstrap will now auto-detect images + masks on the first launch, convert binary masks to YOLO bounding-box labels, split 70/15/15, and write data.yaml.

2. Train and export the model#

Once the dataset is in the right place, the repo script will download the YOLO11n base model, train it, export it, and cache the final OpenVINO IR.

Model references:

  • YOLO11n model family: https://docs.ultralytics.com/models/yolo11/

  • Ultralytics detect task: https://docs.ultralytics.com/tasks/detect/

  • Base weights (~5.4 MB, downloaded automatically during make backend-bootstrap): https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11n.pt

If you already have a trained IR, you can seed it into models/ and skip the training entirely. The presence of both models/yolo11n_polyp/best_openvino_model/best.xml and models/yolo11n_polyp/.trained_ok short-circuits the bootstrap to ready in seconds.

make assets   # copies best.xml + best.bin from poc/st2_app, if present

2.1 Optional: pre-place yolo11n.pt when GitHub is blocked#

If downloading yolo11n.pt fails during bootstrap (for example, timeout / curl 28), pre-download it on a machine with GitHub access and copy it into the backend cache:

# On a machine that can access GitHub:
curl -L -o yolo11n.pt \
        https://github.com/ultralytics/assets/releases/download/v8.4.0/yolo11n.pt

# On the target host (container must exist from at least one `make up`):
docker cp yolo11n.pt surgical-backend:/cache/weights/yolo11n.pt
docker restart surgical-backend

The backend checks ${CACHE_DIR}/weights/<model_name>.pt (default /cache/weights/yolo11n.pt) before fallback to auto-download.

2.2 Optional: pre-train and export model before make up#

You can prepare the model artifacts ahead of time:

make backend-venv       # one-time Python environment: torch+xpu, Ultralytics, OpenVINO
make backend-bootstrap  # prepares ColonDB, trains YOLO11n, exports FP16 OpenVINO IR

backend-bootstrap does the full model preparation in sequence:

  • reads the dataset from datasets/ColonDB/raw/

  • extracts the archive if needed

  • creates the train/validation/test split

  • downloads the base yolo11n.pt weights through Ultralytics

  • trains YOLO11n using the settings in backend/config/model.yaml

  • exports the best checkpoint to FP16 OpenVINO IR

  • writes the .trained_ok cache marker

Expected output layout:

models/yolo11n_polyp/
├── .trained_ok
└── best_openvino_model/
        ├── best.xml
        ├── best.bin
        └── metadata.yaml

2.3 Optional: train elsewhere and import the IR#

If you have trained YOLO11n on ColonDB on another machine:

  1. Export the checkpoint to FP16 OpenVINO IR:

    yolo export model=/path/to/best.pt format=openvino half=True imgsz=640

  2. Copy the result into this exact layout:

    Surgical_Instrument/models/yolo11n_polyp/
├── .trained_ok
└── best_openvino_model/
    ├── best.xml
    ├── best.bin
    └── metadata.yaml

    best.xml and best.bin are required. metadata.yaml is recommended because DL Streamer/OpenVINO can use the exported Ultralytics metadata.

  3. Create the .trained_ok cache marker (checked by make up and the backend):

    mkdir -p models/yolo11n_polyp/best_openvino_model
date -Is > models/yolo11n_polyp/.trained_ok

3. Generate demo video (required)#

Fresh clones do not include videos/polyp_test.mp4. Generate it before running make doctor / make up.

.venv-backend/bin/python scripts/create_endoscopy_video.py \
        --images-dir datasets/ColonDB/raw/ColonDB/images \
        --output videos/polyp_test.mp4 \
        --seconds 60 --fps 60 --width 1920 --height 1080

The script creates or overwrites videos/polyp_test.mp4 using an H.264-compatible codec (required by the default file-source pipeline). It tries OpenCV first, then falls back to system ffmpeg (libx264) when needed.

4. Bring the stack up#

Before bringing the stack up, run the preflight check:

make doctor   # checks Docker, accelerators, cached IR, demo video, and port 8080

Then discover the camera serial and the P-core set for your CPU:

make list-cameras   # prints Basler serial(s) -> SOURCE_ARG
make show-cores     # prints the P-core set    -> PIPELINE_GST_CORES

make up supports two image sources, controlled by the REGISTRY flag.

4a. Pull images from registry (default)#

REGISTRY=true (the default) pulls the prebuilt images at TAG (default latest) and starts them with RENDER_GID / VIDEO_GID auto-detected from the host — no local build needed.

# Live Basler camera, tuned for fixed exposure + GPU detection with a live window.
make up SOURCE_KIND=basler SOURCE_ARG=<SERIAL_NUMBER> \
        BASLER_PIXEL_FORMAT=ycbcr422_8 BASLER_FIXED_CAMERA=1 BASLER_EXPOSURE_US=5000 \
        BASLER_THROUGHPUT_LIMIT=400000000 DETECT=1 DETECTION_DEVICE=GPU \
        AUTOVIDEOSINK=true PIPELINE_VIDEO_SINK=autovideosink

# Minimal live Basler camera (P-core pinned, free-running sink).
make up SOURCE_KIND=basler SOURCE_ARG=<SERIAL_NUMBER> \
        PIPELINE_GST_CORES=<P_CORES> PIPELINE_SINK_SYNC=false

# Default file source
make up              # trains YOLO11n on first boot if IR is not cached
make logs            # optional: follow readiness/startup logs

4b. Build images from source#

REGISTRY=false builds every image locally from its Dockerfile (backend = torch+xpu wheels + OpenVINO + Ultralytics, UI = Vite build → nginx, pipeline = DL Streamer + gencamsrc).

# Live Basler camera, tuned and built from source
make up SOURCE_KIND=basler SOURCE_ARG=<SERIAL_NUMBER> \
        BASLER_PIXEL_FORMAT=ycbcr422_8 BASLER_FIXED_CAMERA=1 BASLER_EXPOSURE_US=5000 \
        BASLER_THROUGHPUT_LIMIT=400000000 DETECT=1 DETECTION_DEVICE=GPU \
        AUTOVIDEOSINK=true PIPELINE_VIDEO_SINK=autovideosink REGISTRY=false

# Minimal live Basler camera, built from source
make up SOURCE_KIND=basler SOURCE_ARG=<SERIAL_NUMBER> \
        PIPELINE_GST_CORES=<P_CORES> PIPELINE_SINK_SYNC=false REGISTRY=false

# Default file source, built from source
make up REGISTRY=false     # trains YOLO11n on first boot if IR is not cached
make logs                  # optional: follow readiness/startup logs

The surgical-ui service declares depends_on: surgical-backend: condition: service_healthy, so it will not start listening on :8080 until the backend passes its /api/readiness HEALTHCHECK. The backend healthcheck uses a 45-minute start_period to absorb first-boot training.

Follow first-boot progress#

make logs

Expect to see the FSM walk through:

[boot] state=initializing
[boot] state=checking_cache
[boot] state=downloading_dataset      (skipped if raw/ already populated)
[boot] state=preparing_dataset
[boot] state=downloading_weights      (~5 MB yolo11n.pt)
[boot] state=training                 (~15-25 min, ~50 epochs)
[boot] state=exporting                (Ultralytics → OpenVINO IR)
[boot] state=ready
[server] READY

5. Open the UI#

Note that the UI is health-gated on the backend - the browser tab will not answer until surgical-backend reports /api/readiness → ready. On the first boot this time window is 20–35 minutes while YOLO11n trains (an estimate for ColonDB on the Intel® Arc™ iGPU). Subsequent boots take seconds because the trained IR is cached in ./models/.

Once the backend is healthy the UI starts and answers on http://localhost:8080 (you can override it with make up UI_HOST_PORT=9090). make up and make run also print the LAN URL so you can access it from the same network.

Use the left Config accordion to pick source (file or basler), source argument, and device, then click Start to kick off inference. The right-side KPI blocks begin populating within ~1 second.

See more info in User Interface.

For subsequent boots (IR already cached):

make run

make run requires cached IR files and .trained_ok. If they are missing, prepare or seed the model artifacts first.

make up and make run auto-detect /dev/video* devices and layer in a compose override so cameras are available to the pipeline container.

For runtime adjustments — source and device selection via the Settings modal, video upload, and API endpoints for scripting — see Runtime Configuration.

6. Stop / clean up#

If you want to stop or restart the app, you have the following options:

make down                 # stop + remove containers, keep volumes + IR

make clean                # also drop the surgical-cache named volume + built images

The trained IR model under ./models/ is a bind-mount and survives make clean. Delete it manually to force a full re-train on next boot:

rm -rf models/yolo11n_polyp

Common overrides#

Variable

Default

Meaning

UI_HOST_PORT

8080

Only host-published port.

DETECTION_DEVICE

xpu

Set to cpu on a host without an Arc iGPU.

RENDER_GID / VIDEO_GID

auto

Override if the host has non-standard render/video group IDs.

Example: run the whole stack CPU-only on port 9000:

make up UI_HOST_PORT=9000 DETECTION_DEVICE=cpu

Next: pipeline tuning#

The DL Streamer pipeline can be tuned at launch time with environment variables passed to make up, including:

  • SOURCE_KIND

  • DETECT

  • WATERMARK

  • MINIMAL

  • SCHEDULING_POLICY

  • BATCH_SIZE

  • AUTOVIDEOSINK

Three common shapes are used in practice:

  • File preview

  • Minimal live camera

  • Tuned live inference

Basler camera tuning knobs#

For live Basler sources (SOURCE_KIND=basler), the following variables tune the camera and the display sink. See the tuned command in 4a. Pull images from registry for a full example.

Variable

Default

Meaning

BASLER_PIXEL_FORMAT

ycbcr422_8

Camera pixel format (e.g. ycbcr422_8, bayerbggr).

BASLER_FIXED_CAMERA

0

Set to 1 to disable auto-exposure/auto-gain and apply the fixed values below.

BASLER_EXPOSURE_US

unset

Fixed exposure time in microseconds (applied only when BASLER_FIXED_CAMERA=1).

BASLER_THROUGHPUT_LIMIT

400000000

DeviceLinkThroughputLimit in bytes/sec; lifts the camera off its default cap.

DETECT

1

Set to 0 to run passthrough (no inference).

AUTOVIDEOSINK

unset

true opens a live render window (DISPLAY_VIEW=1 + SINK_SYNC=true); false disables it.

PIPELINE_VIDEO_SINK

autovideosink

GStreamer sink element (e.g. autovideosink, xvimagesink).