Get Started (UAV Mission Compute SDK Mode)#
This guide provides a step-by-step walkthrough for testing the UAV Vision Analytics application in UAV Mission Compute SDK mode and running the demo with a simulated UAV camera feed/RealSense cameras.
How It Works#
A minimal stack. Telemetry is received via MQTT from the uav-mission-compute-sdk project, which must be started first. The DL Streamer Pipeline Server container reads armed/disarmed state from uav/{id}/telemetry/status and subscribes to three RTSP camera streams (nadir, forward, rear).
Telemetry / pipeline lifecycle flow:
sequenceDiagram
participant Client as curl / QGroundControl / ffplay
participant Nginx as nginx (TLS :443, RTSP :8555)
participant DL_Streamer_Pipeline_Server as DL Streamer Pipeline Server
participant SDK as uav-mission-compute-sdk
participant OVL as gvapython (MavlinkReceiver)
participant Frame as Video Frame
Client->>Nginx: POST /pipelines/... (HTTPS :443)
Nginx->>DL_Streamer_Pipeline_Server: proxy_pass REST :8081
DL_Streamer_Pipeline_Server-->>Nginx: 200 instance_id
Nginx-->>Client: 200 instance_id
SDK->>OVL: broadcast MQTT Telemetry :1883
Note over OVL: background thread parses<br/>GLOBAL_POSITION_INT, VFR_HUD,<br/>GPS_RAW_INT into latest_data
Frame->>OVL: process_frame() per frame
OVL->>Frame: ROI labels (ALT · SPD · HDG · LAT · LON · SATS)
Frame->>Nginx: annotated RTSP :8555 (stream {} passthrough)
Nginx->>Client: rtsp://<HOST_IP>:8555/...
Services:
Service |
Image |
Ports |
Role |
|---|---|---|---|
|
|
|
AI inference, RTSP output |
|
|
|
CPU, GPU, NPU, and memory metrics for the ground-station dashboard (the container is not named |
|
|
|
Reverse proxy / TLS termination — the only service that publishes ports to the host |
Caution
dlstreamer-pipeline-server no longer publishes ports directly — it is reachable only
through nginx on the internal app_network.
Steps to Test the Application#
Prerequisites#
Docker and Docker Compose v2
Intel platform with at least 16 GB RAM (Panther Lake recommended)
Network access to pull Docker images (configure proxy if behind a corporate firewall)
The following system packages:
sudo apt install -y python3.12-venv ffmpeg
Note
python3.12-venv is required by make model to create a Python virtual environment.
ffmpeg provides ffplay for viewing the RTSP output stream and ffmpeg for recording.
1. Start the UAV Mission Compute SDK#
There are two options available to get the application source:
Download the compressed file and get into the directory:
curl -OjL https://github.com/open-edge-platform/federal-and-aerospace-ai-suite/releases/download/fedaero-latest/uav-mission-apps.zip
Decompress the downloaded file:
unzip uav-mission-apps.zip
cd uav-mission-compute-sdk/
Clone the repository, get into the directory and start the SDK’s core infrastructure (PX4, MQTT broker, MediaMTX RTSP server).
git clone https://github.com/open-edge-platform/federal-and-aerospace-ai-suite.git --branch main
cd federal-and-aerospace-ai-suite/uav-mission-compute-sdk
Then, for either option, initialize the environment:
make init # create .env, detect GPU
Caution
Follow only Step 0 (configure credentials) and Step 1+2 (make up-sim-camera) from the get-started guide / SDK README. Do not run make apps (SDK Step 3) — that starts the SDK’s own AI vision-processor and dashboard, which is not needed here since uav-vision-analytics runs its own inference via DL Streamer Pipeline Server.
The SDK’s .env defaults to HOST_IP=127.0.0.1, which binds MQTT, RTSP, and all other published ports to loopback only. Since uav-vision-analytics runs in a separate Docker container/network, it cannot reach loopback-bound ports. Set the SDK’s .env to bind on all interfaces before starting it:
sed -i 's|^HOST_IP=.*|HOST_IP=0.0.0.0|' .env
make up-sim-camera # start PX4, MQTT, RTSP server
2. Configure environment#
From the uav-mission-compute-sdk directory, get into the application directory (it sits next to the SDK in both the ZIP and the cloned repository):
cd ../uav-vision-analytics
Then initialize the environment:
make init
make init creates .env from the template and auto-detects your host IP (HOST_IP), your Intel GPU device paths (GPU_DEVICE, GPU_RENDER_DEVICE), Intel NPU (NPU_DEVICE), and RealSense / USB camera (CAMERA_DEVICE_1). It skips if .env already exists. Open and update .env only if HOST_IP is not the address other machines use to reach this host (for example on a host with several network cards).
3. Prepare the models#
Download and export the YOLO11s detection model to OpenVINO™ FP16 IR:
make model
If any camera in configs/cameras-uavsdk.yaml has captioning.enabled: true, also download and convert the VLM that writes the scene captions (InternVL2-1B, OpenVINO™ IR for the CPU and the GPU):
make vlm-model
Both targets show the model license and ask for confirmation; add ACCEPT_LICENSE=y to accept without the prompt.
Tip
See the AI Model guide and the VLM Model guide for model details.
4. Start the uav-vision-analytics application#
The SDK’s core infrastructure (step 1) must already be running.
make uavsdk-up
5. Run a simple mission#
Note
Video streams are not available until the UAV is armed and actively on a mission.
Run the simple UAV mission in a persistent terminal window to keep the simulation active. The following sequence arms the UAV, commands a takeoff to 10 m, holds for 120 seconds, then lands:
curl -X POST http://localhost:8080/action/arm
curl -sf -X POST http://localhost:8080/action/takeoff \
-H "Content-Type: application/json" \
-d '{"altitude": 10}'
sleep 120
curl -X POST http://localhost:8080/action/land
6. Start inference pipelines#
Note
Open a new terminal window and launch the inference pipeline to begin processing the video stream.
Two options are available depending on your use case:
Option A — Managed RTSP output (recommended)#
Runs pipeline_manager.py inside the DL Streamer Pipeline Server container. It watches the SDK’s MQTT status topic uav/<UAV_ID>/telemetry/status ({"armed": true|false}), starts the cameras of configs/cameras-uavsdk.yaml when the drone arms and stops them when it disarms. Annotated frames are served as RTSP on port 8555.
The cameras are RTSP streams from the SDK (rtsp://host.docker.internal:8554/<UAV_ID>/<camera>, UAV_ID from .env, default uav-1). By default, nadir_camera detects on the CPU, forward_camera on the GPU, and rear_camera on the NPU (skipped when the host has no NPU). A camera whose RTSP source is not available is skipped. Edit the file to change the devices, switch off VLM captions, or hide the overlay; it is read again at every arming. It has the same format as for the simulator, see Configure the cameras.
make check-cameras # validate cameras-uavsdk.yaml and show what would run
make start-rtsp # wait for the drone to arm, then start the cameras
make cameras-status # compare the file with what is running
Captions. With captioning.enabled: true on a camera, the captions are published to MQTT topic uav/caption/<camera id> on the
SDK’s broker (mqtt-broker, host port 1883; MQTT_BROKER_PORT in .env overrides it), not on a broker of this stack: the compose file points the pipelines’
MQTT_HOST/MQTT_PORT at host.docker.internal:1883. Watch them with
docker exec mqtt-broker mosquitto_sub -t 'uav/caption/#'. The ground-station dashboard’s VIZ_REMOTE_MQTT_PORT (default 1883) already matches.
Stop the standalone stack first: it publishes the same port 1883 on HOST_IP.
The streams are available only after the drone arms — see Step 7 — View the output stream for the RTSP URLs.
Option B — Manual REST API#
Note
RTSP streams are not available until the UAV is armed. Run a simple mission first (see Step 5: Run a simple mission).
Start a single camera pipeline directly. The UAVSDK mode loads config-uavsdk.json, which has the same pipeline names as the simulator stack (uav_object_detection_<det> and uav_object_detection_<det>_<vlm>) with an RTSP source, taken from the rtsp_url parameter.
# Detection on the CPU for the nadir camera
INSTANCE_ID=$(curl -k -s -X POST \
https://<HOST_IP>/pipelines/user_defined_pipelines/uav_object_detection_cpu \
-H "Content-Type: application/json" \
-d '{
"destination": {"frame": {"type": "rtsp", "path": "uav-nadir-camera", "encoding": "h264"}},
"parameters": {
"rtsp_url": "rtsp://host.docker.internal:8554/uav-1/nadir",
"detection_instance_id": "instcpu_yolo11s",
"overlay_label": "nadir-CPU"
}
}' | tr -d '"')
echo "Instance ID: $INSTANCE_ID"
# Verify it reached RUNNING state (not ERROR)
curl -k -s https://<HOST_IP>/pipelines/${INSTANCE_ID}/status | python3 -m json.tool
If state is ERROR, check the container logs:
docker logs uav-va-dlstreamer-pipeline-server 2>&1 | tail -20
To use another device change the pipeline name (uav_object_detection_gpu, uav_object_detection_npu, or the _<vlm> variants with captioner_model_path and mqtt_topic in parameters) and the detection_instance_id.
Note
https://<HOST_IP>/... is the DL Streamer Pipeline Server REST API, proxied by nginx on
port 443 (self-signed cert; use curl -k to skip verification). Plain HTTP on port 80
redirects to HTTPS. Replace <HOST_IP> with the value auto-detected by make init (see .env).
7. View the output stream#
View with ffplay#
Install ffmpeg first if not present using sudo apt install ffmpeg.
Any of the annotated streams can be viewed with ffplay -rtsp_transport tcp <RTSP_PATH>.
The stream is served over RTSP/TCP only (port 8555 is the only port published). Without
-rtsp_transport tcp, ffplay negotiates UDP and the window stays blank.
ffplay -rtsp_transport tcp rtsp://<HOST_IP>:8555/uav-nadir-camera # nadir camera
ffplay -rtsp_transport tcp rtsp://<HOST_IP>:8555/uav-forward-camera # forward camera
ffplay -rtsp_transport tcp rtsp://<HOST_IP>:8555/uav-rear-camera # rear camera
Capture all the video streams#
Record all three streams to disk with ffmpeg:
ffmpeg \
-rtsp_transport tcp -i rtsp://<HOST_IP>:8555/uav-nadir-camera \
-rtsp_transport tcp -i rtsp://<HOST_IP>:8555/uav-forward-camera \
-rtsp_transport tcp -i rtsp://<HOST_IP>:8555/uav-rear-camera \
-map 0:v -c:v copy nadir.mkv \
-map 1:v -c:v copy forward.mkv \
-map 2:v -c:v copy rear.mkv
The annotated stream includes bounding boxes for detected objects (person, car, bus, truck, bicycle, and other classes) and a live telemetry overlay (GPS, altitude, speed, heading).
Note
Other ways to view the stream:
Leverage versatile streaming media players such as VLC Player to seamlessly handle, manage, and playback the incoming streams with ease and efficiency.
View in the ground-station dashboard (optional)#
The uav-ground-station web dashboard shows every camera stream, the scene captions and the
CPU/GPU/NPU load of this host in one page. Run it on a separate machine from the UAV SDK
companion-compute board:
cd ../uav-ground-station
make up
Then open http://<VIZ_HOST_IP>:8080, enter the UAV host’s IP (<HOST_IP> above) and
press Connect. The camera cards appear once the cameras run (after the drone arms). See
the UAV Ground Station README.
Important
If you do need to run the dashboard on the same host as
uav-mission-compute-sdk (e.g., a single-machine test), it will conflict
on port 8080: the SDK’s own REST API (arm/takeoff/land/goto)
already binds localhost:8080 on that host, which is also
uav-ground-station’s default VIZ_PORT. Before make up, set a
different port in uav-ground-station/.env (make init creates it first):
VIZ_PORT=8081, then open http://<VIZ_HOST_IP>:8081 instead.
The SDK’s own mediamtx (the sim’s camera-streaming container) also
publishes WebRTC on 8889 + ICE on 8189 to the host, the same defaults
uav-ground-station’s own mediamtx (MEDIAMTX_WEBRTC_PORT /
MEDIAMTX_WEBRTC_ICE_UDP_PORT / MEDIAMTX_WEBRTC_ICE_TCP_PORT) uses —
a single-machine test will fail to start with a port is already allocated error unless one side is moved; to solve this, you can add to
uav-ground-station/.env: MEDIAMTX_WEBRTC_PORT=8890 and
MEDIAMTX_WEBRTC_ICE_UDP_PORT=8190 / MEDIAMTX_WEBRTC_ICE_TCP_PORT=8190.
make up shows video, captions, Flight Summary, and per-host metrics only.
For the dashboard’s Search Videos button and natural-language search
over indexed footage, run make up-vss instead of make up (it brings up
everything make up does, plus 8 extra VSS containers) — it needs its own
required .env variables first (MinIO/Postgres credentials, an embedding
model, and a detection model). See VSS search in the UAV Ground Station Get Started guide,
and tune the VSS search results
once it is running.
Stop an individual pipeline (only needed if you started one manually via Option B in Step 6):
curl -k -X DELETE https://<HOST_IP>/pipelines/${INSTANCE_ID}
8. Stop all services#
Stop and remove the uav-vision-analytics stack (also removes named volumes):
make uavsdk-down
Then stop the SDK’s core infrastructure:
cd ../uav-mission-compute-sdk
make down
Pipelines#
UAV Mission Compute SDK Mode (config-uavsdk.json)#
Nine pipelines with the same names as the simulator stack: uav_object_detection_<det> (cpu, gpu, npu) and uav_object_detection_<det>_<vlm> (vlm = cpu or gpu, with captions on MQTT uav/caption/<camera>). Their source is rtsp_url. configs/cameras-uavsdk.yaml picks them:
Camera |
Pipeline |
Source (inside Docker) |
Output RTSP (host) |
|---|---|---|---|
|
|
|
|
|
|
|
|
|
|
|
|
Note
<UAV_ID> is the UAV_ID environment variable (default: uav-1); set another value in .env if your SDK project uses a different vehicle ID. The cameras file uses the placeholder {uav_id}, so nothing else has to change.
All pipelines are auto_start: false — started by the manager (make start-rtsp) or through the REST API.
REST endpoint: POST https://<HOST_IP>/pipelines/user_defined_pipelines/{name} (proxied by nginx over HTTPS to dlstreamer-pipeline-server:8081)
Telemetry Overlay Fields#
Each output frame carries these overlaid fields in the upper-left corner:
Field |
Source MAVLink message |
Description |
|---|---|---|
|
— |
Name passed as argument to the gvapython |
|
— |
Running frame counter |
|
|
Relative altitude (m) |
|
|
Ground speed (m/s) |
|
|
Heading (degrees) |
|
|
Latitude |
|
|
Longitude |
|
|
GPS satellites visible |
Port Reference#
nginx is the only service that publishes ports to the host (bound to HOST_IP, not
0.0.0.0). dlstreamer-pipeline-server is reachable only on the internal app_network.
Port |
Protocol |
Service |
Published to host? |
Mode |
|---|---|---|---|---|
|
HTTP |
|
Yes |
All modes |
|
HTTPS |
|
Yes |
All modes |
|
RTSP |
|
Yes |
All modes |
|
HTTP |
DL Streamer REST API |
No (internal only) |
All modes |
Documentation#
Document |
Description |
|---|---|
Application overview and component block diagrams |
|
Operating the application: QGroundControl, ground station, deployment modes |
|
Changing cameras, models, pipelines, overlay and search tuning |
|
RealSense camera setup and pipelines |
|
Performance benchmarking guide |
|
Makefile target reference |
|
Known issues and resolutions |