Get Started (UAV Mission Compute SDK Mode)#

This guide provides a step-by-step walkthrough for testing the UAV Vision Analytics application in UAV Mission Compute SDK mode and running the demo with a simulated UAV camera feed/RealSense cameras.

How It Works#

A minimal stack. Telemetry is received via MQTT from the uav-mission-compute-sdk project, which must be started first. The DL Streamer Pipeline Server container reads armed/disarmed state from uav/{id}/telemetry/status and subscribes to three RTSP camera streams (nadir, forward, rear).

uav vision analytics sdk

Telemetry / pipeline lifecycle flow:

        sequenceDiagram
    participant Client as curl / QGroundControl / ffplay
    participant Nginx as nginx (TLS :443, RTSP :8555)
    participant DL_Streamer_Pipeline_Server as DL Streamer Pipeline Server
    participant SDK as uav-mission-compute-sdk
    participant OVL as gvapython (MavlinkReceiver)
    participant Frame as Video Frame

    Client->>Nginx: POST /pipelines/... (HTTPS :443)
    Nginx->>DL_Streamer_Pipeline_Server: proxy_pass REST :8081
    DL_Streamer_Pipeline_Server-->>Nginx: 200 instance_id
    Nginx-->>Client: 200 instance_id

    SDK->>OVL: broadcast MQTT Telemetry :1883
    Note over OVL: background thread parses<br/>GLOBAL_POSITION_INT, VFR_HUD,<br/>GPS_RAW_INT into latest_data
    Frame->>OVL: process_frame() per frame
    OVL->>Frame: ROI labels (ALT · SPD · HDG · LAT · LON · SATS)

    Frame->>Nginx: annotated RTSP :8555 (stream {} passthrough)
    Nginx->>Client: rtsp://<HOST_IP>:8555/...
    

Services:

Service

Image

Ports

Role

dlstreamer-pipeline-server

intel/dlstreamer-pipeline-server

8081, 8555

AI inference, RTSP output

metrics-manager (container uav-va-metrics-manager)

intel/metrics-manager

9090 (internal; through nginx at /metrics/stream and /api/v1/metrics/latest)

CPU, GPU, NPU, and memory metrics for the ground-station dashboard (the container is not named metrics-manager, which the SDK’s observability profile uses)

nginx

nginx:1.27-alpine

80 (redirects to 443), 443 (HTTPS, self-signed cert), 8555 (RTSP passthrough) — published to HOST_IP

Reverse proxy / TLS termination — the only service that publishes ports to the host

Caution

dlstreamer-pipeline-server no longer publishes ports directly — it is reachable only through nginx on the internal app_network.


Steps to Test the Application#

Prerequisites#

  • Docker and Docker Compose v2

  • Intel platform with at least 16 GB RAM (Panther Lake recommended)

  • Network access to pull Docker images (configure proxy if behind a corporate firewall)

  • The following system packages:

sudo apt install -y python3.12-venv ffmpeg

Note

python3.12-venv is required by make model to create a Python virtual environment. ffmpeg provides ffplay for viewing the RTSP output stream and ffmpeg for recording.

1. Start the UAV Mission Compute SDK#

There are two options available to get the application source:

Download the compressed file and get into the directory:

curl -OjL https://github.com/open-edge-platform/federal-and-aerospace-ai-suite/releases/download/fedaero-latest/uav-mission-apps.zip

Decompress the downloaded file:

unzip uav-mission-apps.zip
cd  uav-mission-compute-sdk/

Clone the repository, get into the directory and start the SDK’s core infrastructure (PX4, MQTT broker, MediaMTX RTSP server).

git clone https://github.com/open-edge-platform/federal-and-aerospace-ai-suite.git --branch main
cd federal-and-aerospace-ai-suite/uav-mission-compute-sdk

Then, for either option, initialize the environment:

make init                # create .env, detect GPU

Caution

Follow only Step 0 (configure credentials) and Step 1+2 (make up-sim-camera) from the get-started guide / SDK README. Do not run make apps (SDK Step 3) — that starts the SDK’s own AI vision-processor and dashboard, which is not needed here since uav-vision-analytics runs its own inference via DL Streamer Pipeline Server.

The SDK’s .env defaults to HOST_IP=127.0.0.1, which binds MQTT, RTSP, and all other published ports to loopback only. Since uav-vision-analytics runs in a separate Docker container/network, it cannot reach loopback-bound ports. Set the SDK’s .env to bind on all interfaces before starting it:

sed -i 's|^HOST_IP=.*|HOST_IP=0.0.0.0|' .env
make up-sim-camera        # start PX4, MQTT, RTSP server

2. Configure environment#

From the uav-mission-compute-sdk directory, get into the application directory (it sits next to the SDK in both the ZIP and the cloned repository):

cd ../uav-vision-analytics

Then initialize the environment:

make init

make init creates .env from the template and auto-detects your host IP (HOST_IP), your Intel GPU device paths (GPU_DEVICE, GPU_RENDER_DEVICE), Intel NPU (NPU_DEVICE), and RealSense / USB camera (CAMERA_DEVICE_1). It skips if .env already exists. Open and update .env only if HOST_IP is not the address other machines use to reach this host (for example on a host with several network cards).

3. Prepare the models#

Download and export the YOLO11s detection model to OpenVINO™ FP16 IR:

make model

If any camera in configs/cameras-uavsdk.yaml has captioning.enabled: true, also download and convert the VLM that writes the scene captions (InternVL2-1B, OpenVINO™ IR for the CPU and the GPU):

make vlm-model

Both targets show the model license and ask for confirmation; add ACCEPT_LICENSE=y to accept without the prompt.

Tip

See the AI Model guide and the VLM Model guide for model details.

4. Start the uav-vision-analytics application#

The SDK’s core infrastructure (step 1) must already be running.

make uavsdk-up

5. Run a simple mission#

Note

Video streams are not available until the UAV is armed and actively on a mission.

Run the simple UAV mission in a persistent terminal window to keep the simulation active. The following sequence arms the UAV, commands a takeoff to 10 m, holds for 120 seconds, then lands:

curl -X POST http://localhost:8080/action/arm
curl -sf -X POST http://localhost:8080/action/takeoff \
  -H "Content-Type: application/json" \
  -d '{"altitude": 10}'
sleep 120
curl -X POST http://localhost:8080/action/land

6. Start inference pipelines#

Note

Open a new terminal window and launch the inference pipeline to begin processing the video stream.

Two options are available depending on your use case:

Option B — Manual REST API#

Note

RTSP streams are not available until the UAV is armed. Run a simple mission first (see Step 5: Run a simple mission).

Start a single camera pipeline directly. The UAVSDK mode loads config-uavsdk.json, which has the same pipeline names as the simulator stack (uav_object_detection_<det> and uav_object_detection_<det>_<vlm>) with an RTSP source, taken from the rtsp_url parameter.

# Detection on the CPU for the nadir camera
INSTANCE_ID=$(curl -k -s -X POST \
  https://<HOST_IP>/pipelines/user_defined_pipelines/uav_object_detection_cpu \
  -H "Content-Type: application/json" \
  -d '{
    "destination": {"frame": {"type": "rtsp", "path": "uav-nadir-camera", "encoding": "h264"}},
    "parameters": {
      "rtsp_url": "rtsp://host.docker.internal:8554/uav-1/nadir",
      "detection_instance_id": "instcpu_yolo11s",
      "overlay_label": "nadir-CPU"
    }
  }' | tr -d '"')
echo "Instance ID: $INSTANCE_ID"

# Verify it reached RUNNING state (not ERROR)
curl -k -s https://<HOST_IP>/pipelines/${INSTANCE_ID}/status | python3 -m json.tool

If state is ERROR, check the container logs:

docker logs uav-va-dlstreamer-pipeline-server 2>&1 | tail -20

To use another device change the pipeline name (uav_object_detection_gpu, uav_object_detection_npu, or the _<vlm> variants with captioner_model_path and mqtt_topic in parameters) and the detection_instance_id.

Note

https://<HOST_IP>/... is the DL Streamer Pipeline Server REST API, proxied by nginx on port 443 (self-signed cert; use curl -k to skip verification). Plain HTTP on port 80 redirects to HTTPS. Replace <HOST_IP> with the value auto-detected by make init (see .env).

7. View the output stream#

View with ffplay#

Install ffmpeg first if not present using sudo apt install ffmpeg.

Any of the annotated streams can be viewed with ffplay -rtsp_transport tcp <RTSP_PATH>. The stream is served over RTSP/TCP only (port 8555 is the only port published). Without -rtsp_transport tcp, ffplay negotiates UDP and the window stays blank.

ffplay -rtsp_transport tcp rtsp://<HOST_IP>:8555/uav-nadir-camera      # nadir camera
ffplay -rtsp_transport tcp rtsp://<HOST_IP>:8555/uav-forward-camera    # forward camera
ffplay -rtsp_transport tcp rtsp://<HOST_IP>:8555/uav-rear-camera       # rear camera

Capture all the video streams#

Record all three streams to disk with ffmpeg:

ffmpeg \
  -rtsp_transport tcp -i rtsp://<HOST_IP>:8555/uav-nadir-camera \
  -rtsp_transport tcp -i rtsp://<HOST_IP>:8555/uav-forward-camera \
  -rtsp_transport tcp -i rtsp://<HOST_IP>:8555/uav-rear-camera \
  -map 0:v -c:v copy nadir.mkv \
  -map 1:v -c:v copy forward.mkv \
  -map 2:v -c:v copy rear.mkv

The annotated stream includes bounding boxes for detected objects (person, car, bus, truck, bicycle, and other classes) and a live telemetry overlay (GPS, altitude, speed, heading).

Note

Other ways to view the stream:

  • Leverage versatile streaming media players such as VLC Player to seamlessly handle, manage, and playback the incoming streams with ease and efficiency.

View in the ground-station dashboard (optional)#

The uav-ground-station web dashboard shows every camera stream, the scene captions and the CPU/GPU/NPU load of this host in one page. Run it on a separate machine from the UAV SDK companion-compute board:

cd ../uav-ground-station
make up

Then open http://<VIZ_HOST_IP>:8080, enter the UAV host’s IP (<HOST_IP> above) and press Connect. The camera cards appear once the cameras run (after the drone arms). See the UAV Ground Station README.

Important

If you do need to run the dashboard on the same host as uav-mission-compute-sdk (e.g., a single-machine test), it will conflict on port 8080: the SDK’s own REST API (arm/takeoff/land/goto) already binds localhost:8080 on that host, which is also uav-ground-station’s default VIZ_PORT. Before make up, set a different port in uav-ground-station/.env (make init creates it first): VIZ_PORT=8081, then open http://<VIZ_HOST_IP>:8081 instead.

The SDK’s own mediamtx (the sim’s camera-streaming container) also publishes WebRTC on 8889 + ICE on 8189 to the host, the same defaults uav-ground-station’s own mediamtx (MEDIAMTX_WEBRTC_PORT / MEDIAMTX_WEBRTC_ICE_UDP_PORT / MEDIAMTX_WEBRTC_ICE_TCP_PORT) uses — a single-machine test will fail to start with a port is already allocated error unless one side is moved; to solve this, you can add to uav-ground-station/.env: MEDIAMTX_WEBRTC_PORT=8890 and MEDIAMTX_WEBRTC_ICE_UDP_PORT=8190 / MEDIAMTX_WEBRTC_ICE_TCP_PORT=8190.

make up shows video, captions, Flight Summary, and per-host metrics only. For the dashboard’s Search Videos button and natural-language search over indexed footage, run make up-vss instead of make up (it brings up everything make up does, plus 8 extra VSS containers) — it needs its own required .env variables first (MinIO/Postgres credentials, an embedding model, and a detection model). See VSS search in the UAV Ground Station Get Started guide, and tune the VSS search results once it is running.

Stop an individual pipeline (only needed if you started one manually via Option B in Step 6):

curl -k -X DELETE https://<HOST_IP>/pipelines/${INSTANCE_ID}

8. Stop all services#

Stop and remove the uav-vision-analytics stack (also removes named volumes):

make uavsdk-down

Then stop the SDK’s core infrastructure:

cd ../uav-mission-compute-sdk
make down

Pipelines#

UAV Mission Compute SDK Mode (config-uavsdk.json)#

Nine pipelines with the same names as the simulator stack: uav_object_detection_<det> (cpu, gpu, npu) and uav_object_detection_<det>_<vlm> (vlm = cpu or gpu, with captions on MQTT uav/caption/<camera>). Their source is rtsp_url. configs/cameras-uavsdk.yaml picks them:

Camera

Pipeline

Source (inside Docker)

Output RTSP (host)

nadir_camera

uav_object_detection_cpu_gpu

rtsp://host.docker.internal:8554/<UAV_ID>/nadir

rtsp://<HOST_IP>:8555/uav-nadir-camera

forward_camera

uav_object_detection_gpu_gpu

rtsp://host.docker.internal:8554/<UAV_ID>/forward

rtsp://<HOST_IP>:8555/uav-forward-camera

rear_camera

uav_object_detection_npu_gpu

rtsp://host.docker.internal:8554/<UAV_ID>/rear

rtsp://<HOST_IP>:8555/uav-rear-camera

Note

<UAV_ID> is the UAV_ID environment variable (default: uav-1); set another value in .env if your SDK project uses a different vehicle ID. The cameras file uses the placeholder {uav_id}, so nothing else has to change.

All pipelines are auto_start: false — started by the manager (make start-rtsp) or through the REST API.

REST endpoint: POST https://<HOST_IP>/pipelines/user_defined_pipelines/{name} (proxied by nginx over HTTPS to dlstreamer-pipeline-server:8081)


Telemetry Overlay Fields#

Each output frame carries these overlaid fields in the upper-left corner:

Field

Source MAVLink message

Description

Name

—

Name passed as argument to the gvapython

Frame

—

Running frame counter

ALT

GLOBAL_POSITION_INT.relative_alt

Relative altitude (m)

SPD

VFR_HUD.groundspeed

Ground speed (m/s)

HDG

GLOBAL_POSITION_INT.hdg

Heading (degrees)

LAT

GPS_RAW_INT.lat

Latitude

LON

GPS_RAW_INT.lon

Longitude

SATS

GPS_RAW_INT.satellites_visible

GPS satellites visible


Port Reference#

nginx is the only service that publishes ports to the host (bound to HOST_IP, not 0.0.0.0). dlstreamer-pipeline-server is reachable only on the internal app_network.

Port

Protocol

Service

Published to host?

Mode

80

HTTP

nginx → 301 redirect to HTTPS (no application traffic)

Yes

All modes

443

HTTPS

nginx → proxies to dlstreamer-pipeline-server:8081 (REST API); self-signed cert, use curl -k

Yes

All modes

8555

RTSP

nginx → raw TCP passthrough to dlstreamer-pipeline-server:8555

Yes

All modes

8081

HTTP

DL Streamer REST API

No (internal only)

All modes


Documentation#

Document

Description

index.md

Application overview and component block diagrams

how-to-guides.md

Operating the application: QGroundControl, ground station, deployment modes

how-to-customize.md

Changing cameras, models, pipelines, overlay and search tuning

realsense-guide.md

RealSense camera setup and pipelines

benchmark.md

Performance benchmarking guide

makefile.md

Makefile target reference

troubleshooting.md

Known issues and resolutions