Positioning Operations#
Service Shell#
The current service shell runs without PTZ configuration. It connects to MQTT with the dedicated
positioning.auth identity, publishes retained
scenescape/positioning/status/{resolver_id} status at QoS 0, and registers an
offline last will. Its status payload contains only resolver_id, status, and
timestamp; an LWT timestamp describes the MQTT session start. Health becomes ready
only after a successful broker connection.
Use make positioning to build the runtime image and make init-secrets to create
the positioning.auth broker identity. Start the
service with the Docker Compose controller or tracker profile (make demo /
make deploy), the test Compose stack, or the Helm chart. Production Compose and
Helm mount the CA certificate and auth file read-only.
Compose runs Positioning as the host UID/GID so it can read 0600 ONVIF
credential files. Certificate generation keeps the CA private key at mode 0600;
it is only used on the host to sign certificates.
Positioning connects to the standard broker TLS listener on port 1883, like other
SceneScape services. It verifies the broker with the CA certificate (--rootcert)
and connects with the positioning.auth credentials; no client certificate is
required.
Positioning publishes only its configured
scenescape/positioning/status/{resolver_id} topic and the
scenescape/positioning/pose/{source_id} topics of its assigned sources. Source
ownership is enforced by the Positioning assignment manifest and Resolver
validation at startup; Manager and the broker do not apply Positioning-specific
topic authorization.
flowchart LR
Camera[ONVIF PTZ camera] -->|HTTPS status poll| Poller[Positioning poller]
Poller -->|in-process PTZSample| Resolver[Single Positioning Resolver]
Manager[Manager: home pose and calibration] -->|read-only API + invalidation| Resolver
Resolver -->|TLS 1883: retained pose and status| Broker[MQTT broker]
Broker -->|pose side channel| Controller[Controller consumer: off by default]
Broker -->|pose side channel| Tracker[Tracker consumer: off by default; Phase 5 deferred]
Demo and deploy Compose include Positioning with the controller and tracker
profiles in sample_data/compose/compose.dl-streamer-example.yml. Test stacks that
launch a Controller or Tracker also include it through
tests/compose/compose.positioning.yml.
Source Ownership#
The checked-in configuration contains no camera list. Deployment source ownership is
represented by deployment/positioning-assignments.json. The checked-in manifest is
empty:
{}
When sources are assigned, each source_id maps to exactly one resolver_id; this
manifest does not assign cameras to scenes. The service refuses to start if a
configured source is not assigned to its resolver_id, or if an entry for its
resolver_id is not configured. Manager remains authoritative for camera
calibration and camera-to-scene membership. Source configuration is described in
Positioning Service.
The Scene Controller reads the same manifest to find the Resolver that owns each PTZ camera and subscribes to that Resolver’s status topic; see PTZ Pose Consumption in the Scene Controller. Mount the same deployment manifest into Positioning and the Scene Controller.
Enable ONVIF Access on the Camera#
The Positioning service talks to cameras only through ONVIF over HTTP or HTTPS. It needs the Device, Media, and PTZ services, and optionally the Events service. Settings names differ by vendor; apply these steps on each camera before you probe it:
Enable ONVIF. Many cameras ship with ONVIF disabled, or enable it only after an administrator password is set. Look for ONVIF, Integration protocol, or Network services in the web UI. On Axis, ONVIF settings are under System > ONVIF after the first web user exists. On TP-Link VIGI, look under the network settings for ONVIF.
Create a dedicated ONVIF account for SceneScape. Some vendors keep ONVIF users separate from web UI users (Axis: System > Accounts > ONVIF accounts). Choose the lowest role that allows PTZ status and events, typically Operator. Do not reuse the administrator account.
Note the ONVIF port. It is often 80 or 443, but some cameras use a separate port (TP-Link VIGI uses 2020). The device service path is
/onvif/device_service.Make sure a media profile has a PTZ configuration. The poller uses the first profile that has one, or the profile named by
profile_token.Synchronize the camera clock with NTP. Point the camera at the same NTP source as the SceneScape host (Date and time settings). WS-Security authentication depends on the camera clock; see Troubleshoot ONVIF Access.
Keep the camera’s security features enabled. The poller works with replay attack protection and brute-force protection on (verified on AXIS M5526-E). Use HTTPS (
"scheme": "https", the default) in production and keep certificate verification enabled. Settls_ca_filefor a private camera CA. If a camera has no HTTPS support, explicitly select HTTP only on an isolated network or protected tunnel.Allow network access from the SceneScape host to the camera’s ONVIF port. For move events, the poller uses PullPoint subscriptions, so the camera does not need to reach the host. Host HTTP proxy variables are ignored for camera traffic.
PTZ Camera Onboarding Check#
Before assigning a camera, verify its ONVIF PTZ endpoint with the same client and poller the service uses. The probe prints device identity, the selected PTZ profile, advertised absolute pan/tilt spaces, and optionally event topics. It then polls and reports achieved rate, interval, timestamp uncertainty, position range, and the reported move-status mix:
docker run --rm --network host --user "$(id -u):$(id -g)" \
-v "$PWD/positioning/.local/secrets/<camera>.json:/run/secrets/onvif/<camera>.json:ro" \
--entrypoint python3 intel/scenescape-positioning:$(cat version.txt) \
-m positioning.onvif.probe --host <camera-host> --port <onvif-port> \
--credentials-file /run/secrets/onvif/<camera>.json --list-topics --duration 30
Add --samples to print each PTZSample as JSON. Add
--move-event-topic PTZController/Move to subscribe to a vendor move event. Read
credentials from a file or from ONVIF_USERNAME/ONVIF_PASSWORD; the probe does not
accept them as command-line arguments. Compare the reported space URI and ranges with
the camera’s ptz_calibration.position_space. If reported_move_status is always
IDLE while the camera moves, configure move_status as none or event.
Keep local test configurations and credentials under positioning/.local/, which is
git-ignored. Never place camera endpoints or credentials in tracked files.
Configure Your Own PTZ Camera Locally#
Use the git-ignored positioning/.local/ directory for local test cameras. A complete
setup looks like this:
positioning/.local/
├── positioning.json # service config with your sources
├── positioning-assignments.json # source_id -> resolver_id
├── compose.override.yml # mounts the files above into the container
└── secrets/
└── <source_id>.json # ONVIF credentials, mode 0600
Create the directory and a credentials file for each camera. Use the ONVIF account from Enable ONVIF Access on the Camera.
mkdir -p positioning/.local/secrets && chmod 700 positioning/.local/secrets cat > positioning/.local/secrets/ptz-cam-1.json <<'EOF' {"username": "<onvif-user>", "password": "<onvif-password>"} EOF chmod 600 positioning/.local/secrets/ptz-cam-1.json
Edit the file in an editor rather than typing the password into shell history.
Verify each camera with the probe (see PTZ Camera Onboarding Check). Note the ONVIF port, the profile token if you need a specific one, whether
reported_move_statuschanges toMOVINGwhile the camera moves, and whether the event topic list contains a move event.Create
positioning/.local/positioning.json. Each key undersourcesis the Manager camera UID. This example has one camera that only supportsGetStatusand one that also publishes an Axis-styleis_movingevent:{ "resolver_id": "positioning-dev", "poll_interval_s": 0.2, "max_pose_age_s": 0.5, "max_timestamp_uncertainty_s": 0.01, "max_clock_offset_s": 0.05, "target_reprojection_rms_px": 3.0, "sources": { "ptz-cam-1": { "type": "onvif_ptz", "host": "<camera-1-ip>", "scheme": "http", "port": 2020, "credentials_file": "/run/secrets/onvif/ptz-cam-1.json", "move_status": "get_status" }, "ptz-cam-2": { "type": "onvif_ptz", "host": "<camera-2-ip>", "scheme": "http", "port": 80, "credentials_file": "/run/secrets/onvif/ptz-cam-2.json", "move_status": "event", "move_event": { "topic": "PTZController/Move", "data_item": "is_moving" } } } }
The four Resolver limits are required whenever
sourcesis not empty; the service exits at startup if any is missing.max_pose_age_smust be at leastpoll_interval_s + max_timestamp_uncertainty_s. See Configure for their meaning.These examples explicitly use HTTP for isolated test cameras. Production configurations should use HTTPS; HTTP is only appropriate on an isolated network or protected tunnel.
Choose
move_statusper camera:Camera behavior during the probe
Setting
GetStatusreportsMOVINGwhile moving"move_status": "get_status"GetStatusalways reportsIDLE, a move event is available"move_status": "event"plusmove_eventNeither is reliable
"move_status": "none"You want event records for calibration only
Keep
get_statusornoneand addmove_eventcredentials_fileis the path inside the container, not on the host.Assign every configured source to the same
resolver_idinpositioning/.local/positioning-assignments.json:{ "ptz-cam-1": "positioning-dev", "ptz-cam-2": "positioning-dev" }
Create
positioning/.local/compose.override.yml. It supplies the configuration, manifest, and credentials to thepositioningservice, which connects to the standard broker TLS listener on port 1883.Compose replaces a whole
commandorentrypointlist, so repeat every option. Without--ntp, clock synchronization cannot be verified and every pose is invalid. Setuserto your host user and group IDs (id -uandid -g) so the service can read the0600credential files:services: positioning: user: "1000:1000" command: - --broker - broker.scenescape.intel.com - --brokerauth - /run/secrets/positioning.auth - --rootcert - /run/secrets/certs/scenescape-ca.pem - --ntp - ntpserv - --config - /run/positioning/positioning.json - --assignments - /run/positioning/positioning-assignments.json volumes: - ./positioning/.local/positioning.json:/run/positioning/positioning.json:ro - ./positioning/.local/positioning-assignments.json:/run/positioning/positioning-assignments.json:ro - ./positioning/.local/secrets:/run/secrets/onvif:ro
The secret files mounted by the base Compose file (broker auth and CA certificate) must also be readable by the user ID you set.
Pose consumers read the same manifest. To let the Scene Controller consume poses, add a
sceneservice to this file as described in PTZ Pose Consumption in the Scene Controller.Start the demo stack, then apply the override.
make demogenerates the rootdocker-compose.ymland starts Positioning with the default configuration, which has no sources:SUPASS=<password> make build-core demo
From the repository root, recreate Positioning with the override and follow the service logs:
COMPOSE="docker compose -f docker-compose.yml -f positioning/.local/compose.override.yml --profile controller" $COMPOSE up -d --force-recreate positioning docker compose logs -f positioning
Use
--profile trackerinstead if the stack runs the Tracker profile. Run the same commands again after you edit any file inpositioning/.local/; the files are read only at startup.On startup, Positioning publishes
onlineonscenescape/positioning/status/<resolver_id>.
At startup the service logs Configured PTZ sources: [...] and
PTZ source <id> connected: ... for each camera. Default INFO logs show a short
Camera <id>: ready (stationary) or Camera <id>: unavailable (<reason>)
message only when the resolved state or calibration version changes. Unchanged
samples and periodic metrics do not produce INFO entries. Warnings and errors
remain visible.
The calibration=... suffix identifies the PTZ movement-model version loaded
from Manager’s camera.ptz_calibration.calibration_version. It is chosen by
the operator or export tool, not generated by Positioning, and is separate from
the static AprilTag/manual reference pose. It does not indicate when the latest
UI calibration was saved or prove current projection accuracy. See
Understanding the Model Version
and the ideal-camera example.
Add --log-level DEBUG to the Positioning service command to include the
structured ptz_move_event records and full positioning_metrics snapshots
every 60 seconds. Metrics collection and MQTT pose publication are unchanged
by log verbosity. Configuration errors,
such as a source missing from the manifest, unknown keys, or an unreadable
credentials file, stop the service with a message that names the source.
For Helm, set positioning.logLevel=DEBUG in your deployment values (or use
helm upgrade --set positioning.logLevel=DEBUG). Return to INFO after
diagnostics if the extra event and metrics records are not needed.
To return to the default service shell, recreate the service without the override file:
docker compose --profile controller up -d positioning
If pgserver fails at startup because it found PostgreSQL data from another
checkout, the checkouts share Compose volumes, which are prefixed scenescape_ by
default. Set a distinct COMPOSE_PROJECT_NAME for this checkout before
make demo, and pass the same value to the docker compose commands above.
Troubleshoot ONVIF Access#
Start with the probe (see
PTZ Camera Onboarding Check). It reports the same
errors as the service, and its clock_offset_s field shows the camera clock minus
the host clock.
Clocks not synchronized#
ONVIF uses WS-Security UsernameToken digest authentication. The token includes a
Created timestamp, and cameras reject tokens outside a short window around their
own clock, often about 5 seconds. Replay attack protection also rejects reused
nonces. Without clock compensation, a camera whose clock differs from the host
rejects correct credentials with:
SOAP fault ...:Sender/ter:NotAuthorized: Sender not authorized
The poller handles this. Before authenticating, it reads the camera clock with an
unauthenticated GetSystemDateAndTime call and offsets Created by the measured
difference. After any NotAuthorized fault, it reads the camera clock again and
retries once. A camera clock that is off by minutes or years therefore still
authenticates. Do not disable replay protection to work around clock skew; other
ONVIF tools that do not compensate for skew may fail where the poller succeeds.
You may still see clock-related failures in these cases:
The camera clock drifts quickly. A camera without NTP can drift about 7 seconds per hour. Each time the drift exceeds the camera’s window, one request fails, the clock is resynchronized, and the retry succeeds. That sample has a larger timestamp uncertainty. Enable NTP on the camera to avoid this.
The camera returns an unreadable time. The service reports
Malformed GetSystemDateAndTime response. Set the camera time manually or through NTP.The host clock is wrong. Samples are timestamped with the host clock, not the camera clock. Check that the host is synchronized:
timedatectl | grep -i synchronized # expect: System clock synchronized: yes
To read the camera clock without credentials:
curl -s --noproxy '*' -H 'Content-Type: application/soap+xml' \
-d '<s:Envelope xmlns:s="http://www.w3.org/2003/05/soap-envelope"><s:Body><GetSystemDateAndTime xmlns="http://www.onvif.org/ver10/device/wsdl"/></s:Body></s:Envelope>' \
http://<camera-host>:<onvif-port>/onvif/device_service
The UTCDateTime element in the response is the camera’s UTC time.
Other common errors#
Message in probe or service logs |
Likely cause and fix |
|---|---|
|
Wrong username or password, or a web UI account instead of an ONVIF account. Check the ONVIF user list on the camera. |
|
The camera requires HTTP Digest and rejected the credentials. Verify the account and its role. |
|
Brute-force protection temporarily blocked the host IP. Wait a few minutes before retrying; the service backs off automatically. |
|
Wrong host or port, ONVIF disabled, or a firewall in between. Check the port from the host, for example with the |
|
ONVIF PTZ is not enabled, or the camera is not a PTZ model. |
|
No media profile has a PTZ configuration, or |
|
The camera has no free PullPoint subscriptions. Cameras limit concurrent subscriptions (Axis: 4), and other clients may hold them. Stop other ONVIF event clients or restart the camera. |
Samples with NaN ranges |
The position space reported by |
Shadow Validation Artifacts#
When the Controller and Tracker shadow profiles run, each emits sampled,
structured JSONL records to stdout only. The Controller’s record format is in
PTZ Pose Consumption; the Tracker
profile is not implemented yet. Records share a run_id; the acceptance
harness combines both profiles by that ID into the immutable
tests/artifacts/ptz/shadow/<run_id>.jsonl artifact and publishes it with the CI or
acceptance run. Services do not write local artifact files or require production
artifact volumes. Field runs export an equivalently named bundle from structured logs.
Run the deterministic Positioning replay and deployment checks with:
pytest -q tests/sscape_tests/positioning
These tests exercise reconnect/backoff, poller and Resolver session changes, sequence gaps, retained stale-pose admission, calibration invalidation/reload, settling, and pose math.
Hardware Acceptance Evidence#
tests/data/ptz/reprojection_golden.json is a held-out hardware measurement of the
Positioning service: the PoseContexts it published for an AXIS M5526-E at 56 stops
(pan, tilt and all four dual-axis directions, each approached from both sides, two
repetitions), scored against AprilTags detected after each stop. It contains the
Manager calibration and home pose, each stop’s ONVIF trace, the published
PoseContext, and the tag correspondences; no images or credentials.
tests/data/ptz/acceptance_profile.json holds the budgets per MOT profile.
test_positioning_hardware_golden.py recomputes the accuracy metrics with the
Controller’s projection code, replays the ONVIF traces through the production
Resolver (which must reproduce every published pose exactly and never validate a
moving sample), and checks the results against the profile:
Metric ( |
Budget |
Measured |
|---|---|---|
Ground-plane error, p95 / max |
0.25 m / 0.25 m |
0.031 m / 0.045 m |
Reprojection error, p95 / max (2688x1512) |
20 px / 25 px |
14.9 px / 18.4 px |
PoseContext latency (sample to MQTT receipt), p95 / max |
0.1 s / 0.2 s |
0.026 s / 0.052 s |
Post-motion availability after mechanical stop, p95 / max |
< 1 s / < 1 s |
0.69 s / 0.71 s |
Valid PoseContexts sampled while moving |
0 |
0 |
The static home calibration alone measures 14.7 px p95, so PTZ motion adds little
error. The budgets apply to poll_interval_s 0.05 and move_status: get_status;
on this camera the PTZController/Move event reported the stop about 0.5 s late,
which raised post-motion availability to 1.13 s mean. Lower poll_interval_s also
raises the Controller’s parked-camera frame admission (see
PTZ Pose Consumption): 20% at 0.2 s,
40% at 0.1 s and 68% at 0.05 s for a pipeline with about 40 ms detection latency.
Size sample_ring_buffer_size for the shorter interval.
These values apply to the measured camera and configuration only. Before enabling a consumer for another camera, firmware, calibration, or Positioning configuration, repeat the same held-out measurement on that hardware and review the budgets. The measurement does not yet cover MOT track continuity with people in the scene, Tracker consumption, or the full ONVIF travel range. Do not enable a consumer in production until its own profile passes.
Live recovery checks on the same deployment (camera parked, Controller enforcing,
observer on the regular broker listener) all failed closed: no PoseContext sampled
before or during an outage was received as valid afterwards, and the Controller
rejected frames as sample_stale until fresh poses returned.
Scenario |
Observed |
First fresh valid pose |
|---|---|---|
Positioning restart |
Publishing stops at SIGTERM; status |
1.9 s after start |
Broker restart |
All clients reconnect; Controller resubscribes |
0.9 s after restart |
Adapter stall (3 s, no samples) |
No poses; continuity gap restarts settling |
0.6 s after resume |
Positioning stopped, late subscriber |
Retained pose already past |
2.0 s after start |
Positioning stops publishing poses immediately on SIGTERM, but its retained offline
status follows about 5.6 s later while workers shut down; consumers rely on
valid_until rather than that status. Calibration invalidation and reload are covered
by the software replay tests above.
Deployments and Rollback#
Demo, deploy, and Compose tests include the Positioning service with the Scene
Controller or Tracker profiles. Helm installs it when positioning.enabled=true; it
has one replica and uses a Recreate rollout to prevent old and new resolver pods
from publishing simultaneously for the same source. This preserves single-writer
ownership at the cost of a brief interruption during upgrades. Do not scale the
deployment above one replica; the service has no leader election.
The Controller PTZ modes, CONTROLLER_PTZ_ENABLED and CONTROLLER_PTZ_SHADOW_MODE
(Helm: scene.ptz.enabled, scene.ptz.shadowMode), are false by default and
mutually exclusive. Keep both off until the Controller’s shadow gate passes. The
Tracker does not consume poses yet.
When Positioning sources are configured in Helm, set
positioning.networkPolicy.cameraCIDRs to the exact camera address ranges and
positioning.networkPolicy.cameraPorts to the ONVIF TCP ports. The chart denies
all Positioning ingress and restricts egress to DNS, Manager HTTPS, the broker TLS
listener (port 1883), NTP, and those camera addresses. Helm fails closed if sources
are configured without camera CIDRs. Docker Compose does not provide equivalent egress ACLs;
apply the same restrictions with the host firewall or deployment network policy.
For Helm ONVIF credentials, prefer a pre-created Kubernetes Secret and set
positioning.onvifCredentials.existingSecret to its name. Store one key named
<source_id>.json per camera; each file contains its ONVIF username and password.
For example:
kubectl create secret generic positioning-onvif \
--from-file='<source_id>.json=./<source_id>.json' -n <NAMESPACE>
helm upgrade scenescape kubernetes/scenescape-chart -n <NAMESPACE> \
--reuse-values --set positioning.onvifCredentials.existingSecret=positioning-onvif
Do not put production credentials in Helm values: Helm release records retain
provided values, even when a chart renders them into a Kubernetes Secret. The
chart’s positioning.onvifCredentials.sources option is intended for disposable
development installations. After changing or rotating a mounted ONVIF Secret,
increment positioning.onvifCredentials.rolloutRevision in the Helm upgrade so
the pollers restart and load the new credentials. The pod template records only
this caller-supplied revision, not credential data or a credential-derived hash.
The service publishes resolved poses but does not alter detections itself. The Scene
Controller consumes poses only when its PTZ mode is enabled; Tracker does not consume
them yet. Roll back PTZ polling and pose publishing by removing sources from the
deployment configuration, or disable the Positioning deployment. To return the
Controller to static projection, disable its PTZ mode; see
PTZ Pose Consumption.