Viewing Re-identification Match Latency Metrics#

The controller exports metrics over OTLP/gRPC — a push protocol, not an HTTP endpoint you can curl or scrape. You need something on the receiving end that implements the OTel MetricsService to see them.

Prerequisites#

Network reachability between the controller container and wherever your receiver runs
CONTROLLER_ENABLE_METRICS=true and CONTROLLER_METRICS_ENDPOINT set (typically in your .env file, picked up by docker-compose.yml’s x-controller-base anchor)

Recommended: run the real OTel Collector#

For anything beyond a one-off “is this metric being emitted at all” check, run an actual OpenTelemetry Collector feeding Prometheus + Grafana (or your OTLP-compatible backend of choice) — see otel-collector-config.yaml for a minimal starting config. Point CONTROLLER_METRICS_ENDPOINT at the Collector’s OTLP/gRPC receiver, then query/dashboard from Prometheus as normal.

This is the only path that gives you real percentile queries (see Computing P95/P99 below) — a print-and-discard receiver can’t compute those, only a backend with the histogram’s bucket data over a time range can.

Quick ad-hoc check (no repo changes)#

If you just want to confirm data is flowing before standing up a full Collector, a few lines of Python stood up locally will do it — this isn’t a script shipped in this repo, just something you can save and run yourself:

# save locally as e.g. quick_otlp_check.py -- not part of this repo
# requires: pip install grpcio opentelemetry-proto
from concurrent import futures
import grpc
from opentelemetry.proto.collector.metrics.v1 import metrics_service_pb2, metrics_service_pb2_grpc

class Printer(metrics_service_pb2_grpc.MetricsServiceServicer):
  def Export(self, request, context):
    for rm in request.resource_metrics:
      for sm in rm.scope_metrics:
        for m in sm.metrics:
          print(m.name)
    return metrics_service_pb2.ExportMetricsServiceResponse()

server = grpc.server(futures.ThreadPoolExecutor(max_workers=4))
metrics_service_pb2_grpc.add_MetricsServiceServicer_to_server(Printer(), server)
server.add_insecure_port("[::]:4317")
server.start()
print("Listening on :4317, Ctrl-C to stop")
server.wait_for_termination()

Point CONTROLLER_METRICS_ENDPOINT at <host-or-container-name>:4317, restart the controller, and trigger some tracking activity. You should see metric names printing as they export (every CONTROLLER_METRICS_EXPORT_INTERVAL_S seconds, default 60 — set it lower, e.g. 5, while testing so you’re not waiting a full minute per export).

This is enough to confirm the pipe is connected. For anything you’d act on (latency trends, correlating against camera/tracked-object count), use the real Collector path above.

Metrics reference#

Metric	Kind	Unit	Tagged with	Meaning
`scenescape_controller_reid_match_latency`	histogram	s	`category`	Raw per-match latency samples; source for rigorous P95/P99 downstream
`scenescape_controller_reid_rolling_avg_match_latency`	gauge	s	`category`	Average over the last 10 matches
`scenescape_controller_reid_rolling_min_match_latency`	gauge	s	`category`	Min over the last 10 matches
`scenescape_controller_reid_rolling_max_match_latency`	gauge	s	`category`	Max over the last 10 matches
`scenescape_controller_reid_current_camera_count`	gauge	count	`category`	Cameras configured and confirmed producing embeddings
`scenescape_controller_reid_tracked_object_count`	gauge	count	`category`	Active tracked objects/persons for one category
`scenescape_controller_reid_total_tracked_object_count`	gauge	count	—	Sum of tracked-object count across every category

All ReID gauges and the histogram carry the same category attribute (e.g. person vs car) so multiple tracked categories don’t collide on the same series — except the total tracked-object count, which is intentionally unattributed since it’s already summed across categories.

Plus the pre-existing controller metrics (MQTT throughput, tracking duration, time-chunking stats) — those export the same way, unrelated to ReID.

Computing P95/P99#

The histogram itself doesn’t carry a precomputed percentile — only raw count/sum/min/max plus bucket data. Real P95/P99 come from a proper metrics backend querying that bucket data over a time range, e.g. in Prometheus:

histogram_quantile(0.95,
  sum(rate(scenescape_controller_reid_match_latency_bucket[5m])) by (le, category))

histogram_quantile(0.99,
  sum(rate(scenescape_controller_reid_match_latency_bucket[5m])) by (le, category))

Troubleshooting#

No data reaching the Collector/receiver at all: confirm CONTROLLER_ENABLE_METRICS=true and CONTROLLER_METRICS_ENDPOINT is set — if the endpoint is empty, metrics init silently disables itself even with the flag on. Also confirm the controller can actually reach the receiver’s host:port (same Docker network, correct alias/hostname, port not typo’d).
Other metrics appear but ReID-specific ones (reid_*) never show up: those only fire on real match decisions — confirm cameras are actually streaming and objects are being detected/tracked, not just that the controller is up.
Changed CONTROLLER_METRICS_EXPORT_INTERVAL_S and nothing changed: it’s only read once, at process start (metrics.init()) — restart the scene/controller-analytics container after changing it.