Scene Controller Data Formats#
Message Formats Overview#
Message Format |
Direction |
MQTT Topic |
|---|---|---|
Subscribe |
|
|
Subscribe |
|
|
Publish |
|
|
Point cloud preview (Manager 3D UI) |
Publish |
|
Point cloud preview: the Manager 3D UI requests a live sensor point cloud with
getpointcloudonscenescape/cmd/camera/{camera_id}(same command topic asgetimage). Publishers that support LiDAR/radar visualization reply onscenescape/pointcloud/camera/{camera_id}with a JSON payload (format: xyz_intensity_f32, base64 little-endian float32 points, sensor-local frame). The Scene Controller does not consume this topic.
The Scene Controller only tracks objects and publishes unregulated per-category output. Sensor
correlation, regulated (rate-controlled) output, and region/tripwire event publishing are owned
by the Analytics microservice — see
Analytics Service Data Formats for those message formats
(scenescape/data/sensor/{sensor_id}, scenescape/regulated/scene/{scene_id},
scenescape/event/region/{scene_id}/{region_id}/{event_type}, scenescape/event/tripwire/{scene_id}/{tripwire_id}/{event_type}).
Camera Input Message Format#
The Scene Controller subscribes to the MQTT topic scenescape/data/camera/{camera_id} and
receives detection metadata from perceptual pipelines. The topic and Manager Cam type are
named historically for cameras, but the same contract applies to any perceptual modality
registered as a Cam and publishing detections here (RGB, depth, LiDAR or radar detectors
that emit the same object metadata, and similar sensing that uses pose plus optional
intrinsics). Environmental / singleton sensors use a different topic and do not carry
pose or optics. Messages are validated against the detector definition in
metadata.schema.json.
Top-Level Message Fields#
Field |
Type |
Required |
Description |
|---|---|---|---|
|
string |
Yes |
Camera identifier; must match the |
|
string (ISO 8601 UTC) |
Yes |
Acquisition time of the frame |
|
object |
Yes |
Category-keyed map; each value is an array of detections (e.g. |
|
number ≥ 0 |
No |
Camera framerate (frames per second) when the message was produced |
|
array of string |
No |
Sub-detection labels run on this frame (e.g. |
|
object |
No |
Camera intrinsic parameters ( |
|
array or object |
No |
Sensor dimensions, |
|
object |
No |
Lens distortion coefficients keyed by name ( |
|
object |
No |
Per-frame rigid camera pose for unlocked cameras ( |
Mobile cameras may include per-frame extrinsics. Structurally invalid payloads
fail detector schema validation. Poses that pass the schema but fail semantic
checks (non-finite values, zero-norm quaternion) are logged and ignored. The
observation pose and optics are used for detections, source-camera visibility,
and projected bounds without overwriting registered camera calibration.
The persisted Cam.pose_locked controls whether incoming detector-message
extrinsics and optics are used for that sensor, regardless of physical modality.
When unlocked (pose_locked is false, the default for new cameras), Controller
and Analytics share one apply path: MQTT extrinsics and MQTT intrinsics /
distortion / resolution are ephemeral runtime overrides (never PATCH
Manager). Missing or invalid fields keep the previous observation; accepted live
extrinsics stamp pose_provenance.class of live (optics-only updates do not).
pose_provenance covers pose/extrinsics only — there is no intrinsics or optics
provenance field (see ADR 18).
Live pose and its MQTT optics are retained across configuration refreshes while
unlocked. Before the first valid observation, stored calibration initializes the
pose. When locked, incoming MQTT extrinsics and optics are ignored; configuration
refreshes reload stored calibration.
Locking the pose resets the runtime pose on configuration refresh.
Changing back does not resurrect an earlier observed pose. There is no expiry;
camera removal, scene reassignment, or process restart discards runtime retention.
See Camera Pose Lock for settings,
archive compatibility, and coordinated upgrade requirements. Accepted live
poses are stamped with pose_provenance.class of live (typically
origin: detector.extrinsics) for Analytics regulated camera entries; see
ADR 18 and
Analytics data formats.
Publish pose updates even when object detection is disabled; the server cannot move
a handheld frustum from localization performed only on the device. Pose-only frames
can include empty arrays for the tracked categories, for example
"objects": {"person": [], "car": []}.
Supply full sensor resolution with camera registration and whenever durable
calibrated frame dimensions change via Manager. MQTT resolution/intrinsics do not
persist. For a new registration only, pixel intrinsics without dimensions use a
logged, centered-principal-point estimate (2*cx, 2*cy) rather than silently
retaining the 640x480 model default. Conversely, a create that supplies
resolution (for example [1280, 720] from a demo stream) without cx/cy
stores a principal point at the frame center (width/2, height/2, so 640,
360 for 1280×720) instead of the legacy 320, 240 defaults. This is an
estimate, not a substitute for explicit dimensions with off-center calibration.
Existing registrations are not retrospectively inferred.
The 3D UI refreshes durable optics on database notifications and consumes current
optics from camera metadata and image responses without replacing the camera pose.
The DL Streamer demo Compose stack keeps broker port 1883 internal. External handheld publishers use authenticated MQTT over secure WebSockets through the web service on port 443; no external TCP MQTT port is required.
Detection Object Fields (objects.<category>[*])#
For a 2D box, distance is depth along the optical axis in metres, not radial
range. A normalized bounding_box is already on the normalized image plane:
its coordinates are scaled by depth, not unprojected through intrinsics again.
Only bounding_box_px requires pixel unprojection. Native 3D boxes already supply
their camera-frame geometry and pose; an accompanying distance is ignored for
both bounding-box forms, without changing their placement or size.
For depth-expanded 2D boxes, inferred cuboid dimensions include the observation
pose’s scale. The upright cuboid uses sensor X for width, optical Z for depth,
and sensor Y for height; rotation changes edge directions, not their lengths.
Explicit detection sizes and library size priors remain unchanged.
Field |
Type |
Required |
Description |
|---|---|---|---|
|
string |
Yes |
Object class label (e.g. |
|
object |
One of ① |
Normalized image-space bounding box ( |
|
object |
One of ① |
Pixel-space bounding box ( |
|
array[3] of number |
One of ① |
3D world position ( |
|
array[3] of number |
One of ① |
Geographic position (latitude, longitude, altitude); converted to ECEF internally |
|
array[3] of number |
One of ① |
3D object dimensions ( |
|
number > 0 |
No |
Inference confidence score for this detection |
|
integer ≥ 0 |
Yes ② |
Per-frame detection index |
|
array[4] of number |
No |
Object orientation as a quaternion |
|
number |
No |
Distance from the camera to the detection in metres |
|
array of objects |
No |
Pose keypoints when a pose estimation model is used; each entry: |
|
array of strings |
No |
Flat list of keypoint-name pairs defining connections (e.g. |
|
object |
No |
Semantic attribute bag (see Semantic Metadata Fields) |
① Location constraint: every detection must provide location in exactly one of these forms (enforced by the schema’s
oneOf):
2D image-based:
bounding_boxand/orbounding_box_px(at least one required; both may be present — if so,bounding_boxtakes precedence)3D world-space:
translation+sizeGeographic:
lat_long_alt+size(converted to ECEFtranslationinternally)
② Schema vs runtime: The JSON schema currently lists
idas optional (onlycategoryis in the schema’srequiredarray). However, the controller accessesidunconditionally at runtime and will reject detections that omit it. Always includeidin every detection object.
③ Height reference: with experimental observation processor, the tracker reads
translationas the object’s centre of mass and publishes tracks at their footprint base (zminus half the height).
Semantic Metadata Fields (objects.<category>[*].metadata.<attr>)#
Field |
Type |
Required |
Description |
|---|---|---|---|
|
any |
Yes |
Detected value for this attribute (e.g. |
|
string |
Yes |
Name of the model that produced this attribute |
|
number [0, 1] |
No |
Confidence score for the detected attribute |
Example Camera Detection Message#
The following example shows a typical message published by a camera pipeline (debug fields
omitted; embedding_vector truncated for readability):
{
"id": "atag-qcam1",
"timestamp": "2026-03-26T21:01:31.486Z",
"rate": 10.03,
"objects": {
"person": [
{
"id": 1,
"category": "person",
"confidence": 0.998,
"bounding_box_px": {
"x": 419,
"y": 64,
"width": 192,
"height": 411
},
"keypoints": [
{ "name": "nose", "x": 0.122, "y": 0.157 },
{ "name": "eye_l", "x": 0.115, "y": 0.136 },
{ "name": "eye_r", "x": 0.16, "y": 0.125 },
{ "name": "shoulder_l", "x": 0.262, "y": 0.276 },
{ "name": "shoulder_r", "x": 0.602, "y": 0.198 }
],
"keypoint_connections": [
"nose",
"eye_l",
"nose",
"eye_r",
"eye_l",
"ear_l",
"eye_r",
"ear_r"
],
"metadata": {
"age": {
"label": "39",
"model_name": "age_gender"
},
"gender": {
"label": "Male",
"model_name": "age_gender",
"confidence": 0.979
},
"reid": {
"embedding_vector": "<base64-encoded string>",
"embedding_dimensions": 256,
"model_name": "torch-jit-export"
}
}
}
]
}
}
For the full schema definition, see metadata.schema.json.
External Source Input Message Format#
The Scene Controller subscribes to scenescape/external/{publisher_id}/{thing_type}
(MQTT template parameter name remains scene_id in PubSub APIs). The path id is
always the publisher (configured child scene uid or agent source_id). Scenes
attach via consumer-side bindings, not by addressing a scene inbox. See
ADR 16.
Two payload contracts share the topic, distinguished by source_id:
Configured child scene (no
source_id):{publisher_id}is the sending child’s own id. The controller looks up the child’s configured parent and staticcameraPose. Only scenes with a parent publish this hierarchy form; roots do not emit hierarchy echoes onto the external topic.Unified external source (
source_idpresent):{publisher_id}must equalsource_id. The controller binds the publisher to one or more scenes:Manual:
CONTROLLER_EXTERNAL_SOURCE_BINDINGS=publisher_id:scene_uid,...Geospatial auto-attach (interim):
reference_frame: wgs84attaches to every scene with four-corner geospatial calibration (until a footprint/handoff binder)Cache reuse: pose omitted → scenes that still hold a live cached pose for this publisher
Scene-frame poses: require a manual binding (and
CONTROLLER_TRUSTED_POSITIONING_SOURCESfor acceptance)
Messages with source_id are validated against the external_source definition in
metadata.schema.json.
This section documents the unified external-source payload contract.
To write a converter that maps a source’s native output into this contract and publishes over authenticated MQTT, see Publish Observations from an External Source Adapter.
External Source Top-Level Fields#
Field |
Type |
Required |
Description |
|---|---|---|---|
|
string (ISO 8601 UTC) |
Yes |
Time the observations (and pose, if present) were acquired |
|
string |
Yes |
Publisher id; must match the topic |
|
array |
Conditional |
Source/body-relative observations; required only when |
|
object |
Conditional |
Pose of the source/body origin, used to transform source-level and camera observations into the bound scene (see External Source Pose Fields); may be omitted to reuse the live cached pose |
|
object |
Conditional |
Legacy single-view calibration for pixel detections in top-level |
|
object |
No |
Optional distortion coefficients paired with top-level |
|
array |
No |
Camera configurations and the observations captured by each camera. Each entry has a stable |
|
boolean |
No |
Applies to source-level and all camera-level objects. |
Drone Coordinate and Camera Contract#
Each entry in cameras describes one camera and owns the observations captured by it.
The controller caches each camera’s configuration with the source pose for transform
reuse (internal TTL cache) and projects that configuration onto the platform scene
object’s cameras field for UI frustums. Mounted-camera pixel detections are
localized with a dual transform (platform pose composed with each camera mount)
and, when track is true or omitted, enter Scenescape multi-camera fusion like
ordinary camera input. With track=false, pixel detections are discarded (same as
camera input while tracking is off); use Cartesian translation observations when
the publisher asserts trusted object identity.
With track=false, reusing the same Cartesian object id from different publishers
is an identity collision and the later observation is rejected.
For a drone using this contract:
poseplaces the drone body origin, not the camera optical origin, in the scene. Use a right-handed body frame: x forward, y left, z up. Adapters must convert native frames (for example MAVLink forward/right/down) before publishing.Top-level
objects[*].translationis a point in the body frame, in metres. If supplied,objects[*].rotationdescribes object-local-to-body orientation.A
cameras[*].idis a stable identifier unique within the source (for examplefront). Controllers attribute pixel detections usingsource_id:camera_id.cameras[*].mountis a rigid transform from the camera optical frame into the body frame. Optical axes are x right, y down, z forward. Its requiredtranslationis the camera optical origin expressed in body metres; its requiredrotationis an active optical-to-body quaternion in(x, y, z, w)order. Publish a unit quaternion; a zero quaternion is rejected. An explicit identity mount means aligned axes, not an automatically forward-facing camera.cameras[*].objects[*].translationis in the camera optical frame, in metres.cameras[*].objects[*].bounding_box_pxis in that camera’s image coordinates.Intrinsics specify angular geometry for pixel localization and for ideal-pinhole UI frustums rendered from the platform object’s projected
camerasfield. Distortion-aware boundaries, ground footprints, and video projection remain follow-on work.
Let T_scene_body be the resolved source pose, and T_body_camera the camera’s fixed mount.
Using homogeneous column vectors:
point_scene = T_scene_body * point_body
rotation_scene = rotation_scene_body * rotation_body_object
T_scene_camera = T_scene_body * T_body_camera
point_scene = T_scene_camera * point_camera
The mounting transform is used for camera-local observations only. It must not be applied
to body-relative top-level object translations. Pixel-box localization uses the composed
T_scene_camera transform shown above.
Keep the drone model’s origin at the body origin and correct asset axis conventions on
the model child, not by modifying telemetry.
For reference_frame: wgs84, lat_long_alt uses degrees and altitude in metres above the
WGS84 ellipsoid, as required by the LLA/ECEF conversion; adapters must convert MSL or
relative-flight altitude before publishing. Rotation still maps body axes directly into
the target scene axes, even for WGS84 positions. Native ENU/NED attitude is not automatically
aligned by geospatial calibration. The same rotation must not be assumed valid for several
scenes with differently oriented axes.
Sources with no cameras array continue to use the source-level observation format; omission
of camera metadata does not imply an identity mount. camera_mount at the external-message
top level is not accepted. Every camera entry contains its own complete mount, intrinsics,
stable id, and observations, so calibration for one camera cannot be applied to another. A
pose update from a source already using cameras must include the complete camera list. Messages
without a pose reuse the live source pose; their camera entries still carry the camera
configuration required to interpret those observations. A non-stale camera list may refresh
cached camera state without extending the source-pose expiry.
The controller converts pixel boxes using the corresponding camera’s intrinsics and distortion, then composes the resolved source pose with that camera’s mount. Camera-local Cartesian translations use the same composed transform; top-level Cartesian translations remain body-relative.
Platform citizenship (no data/source topic)#
Dynamic agents remain first-class through the scene object pipeline
(scenescape/data/scene/... / regulated output), not a parallel retained
scenescape/data/source/... product surface. The controller may keep an
internal pose/intrinsics TTL cache for transform reuse; that cache is not
a consumer MQTT contract.
On every accepted message the controller ensures a platform scene object with
id == source_idat the body origin (translation: [0,0,0]if synthesized). Adapters may publish that object explicitly; otherwise the controller adds it. The platform always merges as a trusted identity (retrack=false), even when message-leveltrackis true or omitted for detections.Pose-only updates (
objects: []) still emit/update the platform object and advance the tracker so departed detections age out.When camera configuration is known (message
cameras[]or cached config, including legacy top-levelintrinsics), the platform object carries acamerasarray of{ id, mount, intrinsics, optional distortion }with observations stripped. The 3D view renders child frustums from that object-graph state (Bar A); there is noscenescape/data/source/...topic.Nested ingest
cameras[]remain the wire encoding for mounts and camera-local observations. Frustum lifetime follows platform object presence on regulated output (age-out / absence), not retained MQTT tombstones.Registered perceptual sensors that should behave as Cams (for example handheld devices) use
scenescape/data/camera/{id}and pose lock / provenance — not the external-source topic family.
External-source pose and optics are ephemeral for every perceptual modality that
publishes on this topic (robots, UWB/RTLS, airborne platforms, LiDAR hubs, and so on):
they live only in the Controller TTL pose cache keyed by (scene_uid, source_id) and
are never written to Manager. That matches the unlocked detector-topic policy for
Cam sensors — runtime override only — even though the stores differ (TTL cache vs
SceneModel observation). Intrinsics provenance is not covered on either path
(ADR 18).
External Source Pose Fields (pose)#
Field |
Type |
Required |
Description |
|---|---|---|---|
|
string |
Yes |
|
|
array[4] of number |
No |
Orientation of the source’s local origin, as a quaternion ( |
|
array[3] of number |
If |
Global position of the source’s local origin (latitude, longitude, altitude in metres) |
|
array[3] of number |
If |
Position of the source’s local origin in scene-local coordinates ( |
|
number > 0 |
No |
Estimated accuracy of the reported position in metres, if known |
|
string |
No |
Informational label for what produced this pose (e.g. |
reference_frame determines how the pose is resolved:
wgs84— A global geopose (e.g. from an onboard GNSS/INS). Requires the target scene to have valid four-corner geospatial calibration (map_corners_lla); the controller convertslat_long_altto scene-local coordinates via the scene’s LLA/ECEF transform. Rejected withscene_georeference_unavailableif the scene is not geo-referenced. Any source may publish this frame.scene— A pose already expressed in the target scene’s local coordinates. This is a privileged frame: only accepted fromsource_ids listed in theCONTROLLER_TRUSTED_POSITIONING_SOURCESenvironment variable (comma-separated list), intended for the Scenescape positioning service. Rejected withuntrusted_scene_poseotherwise. Unset or empty trusts no source (fails closed).
Note: Only position is transformed through the scene’s geospatial calibration for
wgs84poses;rotationis passed through unrotated, matching existing camera-detectionlat_long_althandling. Full ENU-to-scene orientation alignment is future work.
External Detection Object Fields (objects[*])#
Field |
Type |
Required |
Description |
|---|---|---|---|
|
string |
Yes |
Category or class of the observed object (e.g. |
|
array[3] of number |
Conditional |
Position of the object relative to the source’s local origin ( |
|
string |
Conditional |
Source-provided identifier; required for Cartesian observations when |
|
array[4] of number |
No |
Rotation of the object as a quaternion ( |
|
array[3] of number |
No |
Object dimensions ( |
|
number > 0 |
No |
Source-reported confidence for this observation |
|
object |
Conditional |
Pixel-space detection box from a camera-like external source using |
|
object |
No |
Semantic attribute bag; same structure as camera input (see Semantic Metadata Fields) |
Unlike camera detections, size is optional here: a source that cannot estimate an object’s
extent may report a point observation. Point objects (no size) remain eligible for
position-based ROI, tripwire, and sensor-tagging analytics, but are excluded from
volume/occupancy/collision analytics.
Report translation at the object’s centre of mass: with
experimental observation processor,
tracked objects are published at their footprint base (z minus half the height). Objects with
track=false bypass the tracker and keep the reported z.
The top-level source track field determines the tracking mode for detection
objects in the message. When track is true or omitted, detections follow the
normal Scenescape tracking path and the controller assigns the published object
id, so the source message may omit detection objects[*].id. When
track=false, Cartesian detections bypass the kinematic tracker/ReID path and
the published object identity is the source-provided id; pixel
(bounding_box_px) detections are discarded, matching camera input while
tracking is off. The platform self-object (id == source_id) is always treated
as a trusted identity regardless of track.
When an object includes bounding_box_px, it is treated as a camera-style pixel detection rather
than as a source-local Cartesian observation. Do not send both translation and bounding_box_px
on the same object. In that case:
A usable source
poseand top-levelintrinsicsmust be available from this message or from the live(scene_id, source_id)cache entry. If either cannot be resolved, the message is dropped and no objects from it are ingested.Scenescape computes the object location with the same pixel-to-location algorithm used for camera detections.
With
track=trueor whentrackis omitted, the localized detection enters normal Scenescape tracking after the platform/mount dual transform. Withtrack=false, pixel detections are discarded; they do not become trusted-identity tracks.
Pose Caching and Message Ordering#
A resolved pose is cached per (scene_id, source_id) for 30 seconds. Camera intrinsics
(and optional distortion) for pixel detections are stored on the same cache entry and preserved
across pose refreshes until replaced or the entry expires. Subsequent messages from the same
source may omit pose and/or intrinsics to reuse the cached values; a message with a pose
and an empty objects array updates the cache and still materializes the
platform scene object at the body origin. Every successfully resolved message,
including one with zero detections, still advances the bound scene’s tracker for
that object category: an accepted empty update is an empty measurement for
detections (so they age out of the published scene state) while the platform
remains present. Messages with
a timestamp older than the currently cached pose are treated as out-of-order and ignored in
favor of the newer cached transform. If no usable transform can be resolved (no pose supplied and
nothing cached, or the cached pose has expired), the message is dropped without ingesting
objects. Pixel detections equally require usable intrinsics (from the message or cache).
Rejection reasons (logged, not published) include:
Reason |
Cause |
|---|---|
|
No |
|
No |
|
Pixel detections present, no |
|
|
|
Pose update for a multi-camera source omitted the complete |
|
|
|
|
|
|
|
|
Source-Identity Trust for Untracked Objects, with Collision Detection#
When an external source message sets track to false, Cartesian observations
bypass Scenescape’s kinematic multi-object tracker/ReID association and merge
through the already-tracked path used for child scenes with retrack=false.
The source-provided id is used as the published object identity. Pixel
(bounding_box_px) detections in the same message are discarded — the same
behavior as camera input while tracking is off — because there is no
cross-camera identity authority for raw detector boxes.
Two different sources publishing the same source-provided Cartesian id could
silently merge distinct objects. The controller therefore allows only one source to hold a live
claim on that id within a (scene, category) pair. A colliding object is dropped and logged;
other Cartesian objects from the same message are still ingested.
Identity is trusted based on the source_id and id values in the payload, not a
cryptographically verified per-device credential. Scenescape’s current MQTT authentication does
not bind individual publishers to individual source IDs (see
ADR 16).
A publisher that can reach the broker can claim either of these values.
Tracked External Objects Use Controller-Assigned IDs#
When an external source message sets track to true or omits it, each object is processed
as a normal Scenescape detection. Mounted-camera pixel detections are first localized with
the dual transform (platform pose composed with each camera mount), then enter the
kinematic tracker/ReID path. The controller assigns the published track id. On this path,
source-provided objects[*].id is optional and treated only as source metadata for the
incoming observation, not as the final published track identity.
Choosing a source_id (Self-Identification for Agents)#
source_id is not provisioned or registered anywhere in Scenescape ahead of time — unlike a
camera or sensor id, which must match a scene/sensor already configured in the database, an
external source simply announces itself by choosing a source_id string and publishing with it.
For messages using track=false with Cartesian observations, choose a persistent
source_id and per-object id. That id becomes the published identity and must
not collide with another publisher’s live identity. Mounted-camera pixel
detections should use track=true (or omit track) so Scenescape can fuse them
after the dual transform; track=false discards those pixels. For tracked
messages, a stable source-provided object id can improve observation
correlation even though Scenescape assigns the published identity. The
deployer/integrator should choose values that are:
Persistent — stable across process restarts and reboots, so a track’s identity (and any cached pose) is recognized as the same source/object next time it publishes, rather than treated as a brand-new one.
Unique — will not collide with another source’s identifier on the same scene/broker.
Recommended choices, in order of preference:
A hardware-rooted identifier already unique to the device — a serial number, a TPM-backed device UUID, or (for a UWB/RTLS tag) the tag’s own hardware/network ID. This is the strongest option because it is normally immutable and cannot be trivially changed by reconfiguring software.
The primary network interface’s MAC address — a practical, widely available choice for robots, drones, and other networked devices; it is unique per interface and typically stable across reboots. This mirrors the convention already used for sensor identifiers elsewhere in Scenescape (see the Sensor Input Message Format example,
02:42:ac:11:00:05.1).A deployer-assigned static name (for example
"drone-1","forklift-north-3") — acceptable as long as it is provisioned once per physical device and not regenerated on every boot or process restart.
Do not use a randomly generated value (for example a fresh UUID minted at process startup)
as source_id: it defeats pose-cache reuse across restarts. For track=false, do not use a
random or resettable object id, because each restart creates a new published identity or may
reuse a stale identity for a different physical object. Prefer hardware-rooted or MAC-based
identifiers specifically to avoid this.
If a robot or drone reports itself as a tracked object (for example, to visualize the
platform itself in the scene alongside objects it observes), use the same persistent identifier
described above for that object’s id — typically the platform’s own MAC address, serial
number, or device UUID — rather than a value tied to the current process/session.
Example: Agent Publishing a Global Pose and Observations#
{
"timestamp": "2026-03-26T21:01:31.486Z",
"source_id": "drone-1",
"pose": {
"reference_frame": "wgs84",
"lat_long_alt": [37.38688947, -121.96410521, 8.07],
"rotation": [0, 0, 0, 1]
},
"objects": [
{
"id": "track-42",
"category": "vehicle",
"translation": [3.2, -1.4, 0.0],
"confidence": 0.91
}
]
}
Example: Drone with Multiple Cameras#
Each camera carries its own calibration and detections. Omit track (or set
track: true) so each camera’s pixel boxes are dual-transformed into the scene
and fused by Scenescape tracking. Detector-local id values are optional hints
only; they are not cross-camera identity. With track: false, these pixel
detections would be discarded.
{
"timestamp": "2026-03-26T21:01:31.486Z",
"source_id": "drone-1",
"pose": {
"reference_frame": "wgs84",
"lat_long_alt": [37.38688947, -121.96410521, 8.07],
"rotation": [0, 0, 0, 1]
},
"cameras": [
{
"id": "front",
"mount": {
"translation": [0.2, 0, 0.1],
"rotation": [-0.5, 0.5, -0.5, 0.5]
},
"intrinsics": { "fx": 905, "fy": 910, "cx": 600, "cy": 350 },
"objects": [
{
"category": "person",
"bounding_box_px": { "x": 400, "y": 120, "width": 80, "height": 210 }
}
]
},
{
"id": "rear",
"mount": {
"translation": [-0.2, 0, 0.1],
"rotation": [0.5, -0.5, 0.5, 0.5]
},
"intrinsics": { "fx": 820, "fy": 825, "cx": 640, "cy": 360 },
"objects": [
{
"category": "person",
"bounding_box_px": { "x": 700, "y": 130, "width": 76, "height": 206 }
}
]
}
]
}
Example: Positioning Service Publishing a Scene-Local Pose#
Requires source_id (e.g. "positioning-service-1") to be listed in
CONTROLLER_TRUSTED_POSITIONING_SOURCES.
{
"timestamp": "2026-03-26T21:01:31.486Z",
"source_id": "positioning-service-1",
"pose": {
"reference_frame": "scene",
"translation": [5.0, 2.0, 0.0],
"rotation": [0, 0, 0, 1],
"provider": "positioning_service"
},
"objects": [
{ "id": "person-7", "category": "person", "translation": [1.0, 0.5, 0.0] }
]
}
Example: Pose-Only Update and Point-Object Observation#
A pose-only update refreshes the cached transform and still materializes the
platform scene object (id matching source_id) at the body origin:
{
"timestamp": "2026-03-26T21:01:41.486Z",
"source_id": "drone-1",
"pose": {
"reference_frame": "wgs84",
"lat_long_alt": [37.38688947, -121.96410521, 8.07],
"rotation": [0, 0, 0, 1]
},
"objects": []
}
A subsequent message reuses the cached pose and reports a point object (no size):
{
"timestamp": "2026-03-26T21:01:42.486Z",
"source_id": "drone-1",
"objects": [
{ "id": "person-3", "category": "person", "translation": [1.0, 0.5, 0.0] }
]
}
Common Output Track Fields#
Note: Sensor input (
scenescape/data/sensor/{sensor_id}) is no longer consumed by the Scene Controller. Sensor correlation is owned by the Analytics microservice — see Sensor Input Message Format and Singleton Sensor Data for the sensor input message format and how tagged data appears on scene objects.
All Scene Controller output messages include an objects array of tracked objects. Each
tracked object contains the following fields:
Field |
Type |
Description |
|---|---|---|
|
string |
Persistent scene-object identifier; assigned by Scenescape for tracked detections, and may be source-provided or a controller-scoped shared identity for |
|
string |
Object type label; same value as |
|
string |
Object class label (e.g. |
|
number |
Inference confidence of the most recent contributing detection |
|
array[3] of number |
3D world position ( |
|
array[3] of number |
3D object dimensions ( |
|
array[3] of number |
Velocity vector ( |
|
array[4] of number |
Orientation quaternion |
|
array of string |
Camera IDs currently observing this object |
|
array of strings, optional |
External publishers that have contributed observations during this active track’s lifetime. Retained until the track expires; this is provenance, not camera visibility |
|
array of objects |
Pose keypoints propagated from detections when available; each entry uses |
|
array of strings |
Flat list of keypoint-name pairs defining the skeleton edges (e.g. |
|
object |
Map of region/sensor IDs to membership metadata. Never populated by the Scene Controller — added by the Analytics microservice when it enriches this data; see note below. |
|
object |
Map of sensor IDs to timestamped readings ( |
|
number or null |
Similarity/distance value to the matched ReID embedding in the configured vector database; higher-is-better for |
|
string |
Re-ID processing state for the object. One of: |
|
array or absent |
History of UUID reassignments for this track. Each element is |
|
string (ISO 8601) |
Timestamp when the track was first created |
|
object |
Semantic attributes propagated from camera detections; present when visual analytics (e.g. age, gender, Re-ID) are configured. Same attribute structure as camera input. See note below. |
|
object |
Per-camera pixel bounding boxes ( |
|
object or absent |
Association gate geometry for UI visualization. Present when the producer publishes gate parameters. See note below. |
Note on
metadatain track objects: Each attribute follows the structure{label, model_name, confidence?}— identical to Semantic Metadata Fields in camera input. Thereidattribute is a special case: in scene outputreid.embedding_vectoris a 2D float array ([[...numbers...]]), whereas in camera input it is a base64-encoded string.metadatais absent when no semantic analytics pipeline is configured.
Note on keypoint propagation:
keypointsandkeypoint_connectionsare optional pass-through fields from object detections. They are included in output objects when present on the contributing detection data.
External observation provenance:
objects[*].source_idslists external publishers that contributed observations to the active track and is preserved through Analytics regulated and event output. It is separate fromvisibility, which is calculated from configured camera geometry: a drone can report an object even when that object’svisibilityis empty. The set accumulates for the lifetime of the track and is not cleared when one source stops reporting; it does not imply that an object lies inside a source camera’s frustum. Native camera tracks do not receive this field automatically.
External Cartesian observations are transformed independently for each bound scene:
translation is transformed once, and an explicitly supplied object rotation is composed
as rotation_scene_source * rotation_source_object. The fixed camera mount does not
affect body-relative translations. No object orientation measurement is invented when
rotation is omitted. Untracked objects continue to preserve publisher-supplied ids,
subject to the existing identity-collision checks.
Note on
similarity: This field holds the metric value returned by the ReID backend in_distanceand is evaluated by the controller using configured metric semantics. ForCOSINE(normalized vectors with backend IP/DOT), value must be abovesimilarity_threshold; for distance-style metrics such asL2, value must be belowsimilarity_threshold. A value ofnullmeans either the ReID query has not been submitted yet (pending_collection), the query found no match below the configuredsimilarity_threshold(query_no_match), or ReID is disabled (reid_disabled).
Note on
reid_statevalues:
pending_collection: Re-ID embedding collection is in progress; query has not been submitted yet.
query_no_match: Query was submitted but no database match was found.
matched: Query found a database match and the object was re-identified.
reid_disabled: Re-ID is disabled for this object lifecycle (for example due to runtime disablement).
Note on
regions/sensors: These fields are added by the Analytics microservice when it consumesscenescape/data/scene/{scene_id}/{thing_type}and republishes enriched, regulated, and region/tripwire-event output.regionsdefaults to{id: {entered: timestamp}}and gains a livedwellvalue for objects currently inside a region:{id: {entered: timestamp, dwell: seconds}}. Exit records expose the final dwell time separately as{"object": <track>, "dwell": <seconds>}in the top-levelexitedarray. See Regulated Scene Output Message Format for full format details.
Note on
association_window: Produced by the Scene Controller and Tracker service so the Manager 2D/3D UI can draw association gates. Shape depends onassociation.method:
euclidean→{method, shape: "circle", radius_m}whereradius_mismax_radius_m; without it, the Scene Controller’s gate, the averagetracking_radiusof the frame’s objects
position_mahalanobis→{method, shape: "ellipse", semi_major_m, semi_minor_m, angle_rad}from the χ² contour of the track’s predicted XY measurement covariance (S_pred), the XY shadow of the track’s gate: χ²(gate_probability, 3) for a track with measured z, χ²(gate_probability, 2) without; each semi-axis clipped tomax_radius_mso the drawn region stays inside the hard association ceiling
angle_radis the major-axis orientation in the scene XY plane (radians from +X toward +Y), defined modulo π: the ellipse is the same after a half turn. The window has no centre of its own: the UIs draw it at the object’s published (corrected)translation, while the gate is centred on the predicted position. The drawing is exact for a track without a detection in that frame and off by the Kalman correction otherwise. Analytics passes the field through on regulated scene output when present.
Data Scene Output Message Format#
Published on MQTT topic: scenescape/data/scene/{scene_id}/{thing_type}
The Scene Controller publishes unregulated (raw) tracking results, one message per object category per scene publication cycle. Each message contains the current state of all tracked objects of that category.
This is a scene-level snapshot, not a copy of the input publisher envelope. The top-level
source_id, pose, track, intrinsics, distortion, and message cameras fields
are omitted. The platform appears here as a normal scene object (id matching
source_id). When camera configuration is known, that platform object includes a
cameras array (id, mount, intrinsics, optional distortion) used for
object-graph frustum rendering. Per-object contributing publishers are represented
by source_ids.
Data Scene Top-Level Fields#
Field |
Type |
Description |
|---|---|---|
|
string |
Scene identifier (UUID) |
|
string (ISO 8601 UTC) |
Publication timestamp |
|
string |
Scene name |
|
number |
Current scene processing rate in Hz |
|
integer |
Cumulative count of unique detections since scene start |
|
array |
Tracked objects (see Common Output Track Fields) |
Example Data Scene Message#
{
"id": "302cf49a-97ec-402d-a324-c5077b280b7b",
"timestamp": "2026-03-26T20:49:59.642Z",
"name": "Queuing",
"rate": 9.984,
"unique_detection_count": 91,
"objects": [
{
"id": "65d49fa0-a855-46f8-bb41-4e92102c7c47",
"category": "person",
"type": "person",
"confidence": 0.999,
"translation": [2.463, 3.61, 0.0],
"size": [0.5, 0.5, 1.85],
"velocity": [-0.045, 0.012, 0.0],
"rotation": [0, 0, 0, 1],
"visibility": ["atag-qcam1", "atag-qcam2"],
"metadata": {
"age": { "label": "32", "model_name": "age_gender" },
"gender": {
"label": "Male",
"model_name": "age_gender",
"confidence": 0.904
},
"reid": {
"embedding_vector": "<embedding_dimensions-element float array>",
"embedding_dimensions": 256,
"model_name": "torch-jit-export"
}
},
"camera_bounds": {
"atag-qcam1": {
"x": 169,
"y": 4,
"width": 96,
"height": 168,
"projected": false
}
},
"similarity": null,
"reid_state": "pending_collection",
"first_seen": "2026-03-26T20:49:49.339Z"
}
]
}
Note: The example above omits
regionsandsensors, which the Scene Controller never populates on this topic (see Common Output Track Fields above). The Analytics microservice consumes this topic and republishes enriched, regulated (rate-controlled), and region/tripwire-event output — includingregionsandsensorson each object. See Analytics Service Data Formats for thescenescape/regulated/scene/{scene_id},scenescape/event/region/{scene_id}/{region_id}/{event_type}, andscenescape/event/tripwire/{scene_id}/{tripwire_id}/{event_type}message formats.