Telemetry Metrics#
This note explains what the /telemetry endpoint returns, how each metric is computed, and how to interpret the numbers when tuning the Multimodal DataPrep microservice.
Endpoint recap#
Path:
GET /telemetryQuery parameters:
limit(default10, max100) – number of most recent records to return (capped by the server-side retention window).source– optional filter that matches the request path that produced the entry (for example/media/upload).
Response shape:
Sample response:
{
"count": 1,
"items": [
{
"request_id": "1dc48f8c-6ee1-4a5c-8d92-032b5bc5591d",
"source": "/media/upload",
"timestamps": {
"requested_at": "2026-04-06T08:05:51.111006Z",
"completed_at": "2026-04-06T08:08:00.346370Z",
"wall_time_seconds": 127.636318
},
"video": {
"bucket_name": "video-summary",
"video_id": "dp_video_1775462750",
"filename": "input.mp4",
"frame_interval": 1,
"fps": 30.0,
"total_frames": 12552,
"video_duration_seconds": 418.4,
"tags": ["intersection", "night"],
"video_url": "http://:8000/v1/dataprep/media/download?video_id=dp_video_1775462750&bucket_name=video-summary",
"video_rel_url": "/v1/dataprep/media/download?video_id=dp_video_1775462750&bucket_name=video-summary"
},
"config": {
"object_detection_enabled": true,
"detection_confidence": 0.85
},
"counts": {
"stream_id": 0,
"frames_extracted": 12552,
"items_after_detection": 8336,
"embeddings_stored": 20888
},
"pipeline_stats": {
"pipeline_wall_duration": 127.636318,
"pipeline_throughput_fps": 163.652,
"pipeline_concurrency_factor": 2.677,
"pipeline_efficiency_pct": 89.253,
"parallel_efficiency_pct": 99.34,
"decode_pipeline_efficiency_pct": 0.8215,
"detect_pipeline_efficiency_pct": 0.9934,
"embed_store_pipeline_efficiency_pct": 0.8627
},
"stage_duration": {
"frame_extraction_seconds": 104.857776,
"detection_seconds": 126.794084,
"embedding_seconds_total": 80.117875,
"embed_preprocess_time": 49.012821,
"embed_inference_time": 30.771018,
"storage_seconds_total": 29.989046,
"total_wall_seconds": 127.636318
},
"stage_throughput": {
"decode_throughput": 119.705,
"detect_throughput": 98.995,
"embedding_preproc_throughput": 426.174,
"embedding_infer_throughput": 678.821,
"embeddings_throughput": 260.716,
"store_throughput": 696.521,
"pipeline_throughput": 163.652
},
"batches": [
{
"stream_id": 0,
"batch_index": 0,
"input_frames": 64,
"items_after_detection": 18,
"detection_seconds": 0.440378,
"embedding_seconds": 0.405169,
"embedding_preproc_seconds": 0.239543,
"embedding_infer_seconds": 0.163761,
"storage_seconds": 0.269737,
"total_seconds": 1.43656,
"embeddings_stored": 82
},
"<other batch entries omitted for brevity>"
]
}
]
}
Each TelemetryRecord is stored in JSONL under data/telemetry/telemetry.jsonl (or the configured path) and is served verbatim after lightweight normalization so that older float timestamps are converted to UTC ISO-8601 strings.
Metrics Manager live publishing#
Set MM_DATAPREP_METRICS_MANAGER_URL to enable direct, event-driven live
publishing. Immediately after a completed record is stored, DataPrep queues:
{
"name": "dataprep_embeddings_per_second",
"value": 260.716,
"timestamp": 1775462880.34637,
"tags": {
"service": "multimodal-dataprep",
"stage": "embedding"
}
}
for POST /api/v1/metrics/simple. The publisher uses a one-item latest-value
queue and a bounded timeout. Connection failures are retried with capped
backoff, but a newer completion supersedes an older retry. Publishing failures
never fail or delay media ingestion, and the existing JSONL history and
GET /telemetry API remain unchanged.
Live-stream embeddings/second#
File uploads publish dataprep_embeddings_per_second once at the end of the
request, as embeddings_stored / pipeline_wall_duration — a job processed
back-to-back, so its wall time is active processing time and the value
reflects the device’s true ingestion rate.
A live RTSP stream never reaches that end-of-request publish, and its frames
arrive in real time: a 30 fps camera sampled every MM_DATAPREP_FRAME_INTERVAL
frames only feeds a few embeddings per second. Dividing by wall-clock time would
therefore report the camera’s delivery cadence (~7 eps), not the device’s
ingestion rate (hundreds of eps). So live streams are measured differently:
A single process-wide aggregator (
LiveThroughputAggregator) samples every 5 s and publishes one combined gauge for the whole fleet — never a per-stream value, which would make the single gauge flap to the last writer.The denominator is active compute time, not wall time. Each stored batch reports its
detect + embed + storeseconds. Decode is excluded (for a live source that stage blocks on real-time packet arrival), and detection is included — it is genuine pipeline work and is ~0 when object detection is disabled, so the rate adapts automatically to OD on/off.Each tick the gauge is
Σ new embeddings / Σ new active secondsacross all running streams. Because embedding inference is serialized behind a shared lock, summing active seconds approximates the device’s real busy time, so the combined figure is the true fleet ingestion rate (a sum, not a per-stream average that would understate the work).Streams that (re)join the running set are baselined for one tick so an accumulated backlog never spikes the gauge, and paused/stopped/errored streams drop out without producing a negative delta.
Intervals that produced no embeddings publish nothing; the gauge holds its last reported rate until the next burst. There is intentionally no decay — EPS reflects the last observed ingestion rate, while device capacity is covered by the separate RAM/CPU/GPU/NPU gauges.
Metric derivations#
Timestamps#
Field |
Description |
Calculation |
|---|---|---|
|
When the pipeline accepted the request. |
Captured at the start of processing and emitted as a UTC string ( |
|
When the final artifact (embeddings + manifests) was written. |
Same formatting as |
|
End-to-end time the request spent in the pipeline. |
Difference between the completion and request timestamps (falls back to |
Video metadata#
This block mirrors the request that was processed:
bucket_name,video_id,filename, andframe_intervalare copied from the active job. Numerical fields (fps,total_frames,video_duration_seconds) come straight from the frame extractor.video_urlandvideo_rel_urlpoint to the download endpoint for the processed video or stitched preview.
Processing config#
Fields such as object_detection_enabled and detection_confidence are captured from the resolved runtime configuration. parallel_workers and batch_size are also included if configured. All fields reflect the effective configuration (after environment variables, CLI args, and defaults are merged) so operators can correlate telemetry with tuning changes.
Aggregate counts#
Field |
Description |
|---|---|
|
Identifier of the video stream processed (0-indexed). Useful when multiple streams are processed in a single request. |
|
Number of keyframes pulled from the source video before detection. |
|
Crops and frames that survived object detection filters. |
|
Items that were successfully embedded and written to VDMS. This value should match the |
Stage durations#
The stage_duration block reports the cumulative time each stage spent across all processed batches. Because the pipeline is concurrent, individual stage totals can exceed wall_time_seconds.
Field |
Description |
|---|---|
|
Total time spent decoding frames from the video source. |
|
Total time spent running object detection across all batches. |
|
Total time for the full embedding stage (preprocessing + inference) across all batches. |
|
Time within the embedding stage spent on image preprocessing (resize, normalize). |
|
Time within the embedding stage spent on model inference only. |
|
Total time spent writing embeddings to VDMS across all batches. |
|
Same value as |
Pipeline statistics#
The pipeline_stats block provides concurrency and efficiency metrics computed by save_batch_results. All are measured against the true wall-clock interval from the first decode operation to the last store operation.
Field |
Description |
Formula |
|---|---|---|
|
True wall-clock duration of the pipeline (seconds). |
|
|
Overall pipeline throughput in frames per second. |
|
|
How many seconds of work are completed per wall-clock second. Values greater than 1 indicate effective use of concurrency. |
|
|
How efficiently the three concurrent worker threads (decode, detect, embed+store) are utilized. 100 % means all threads are busy the entire time. |
|
|
How well the slowest (bottleneck) stage fills the pipeline wall time. 100 % means the bottleneck stage ran continuously without idle gaps. |
|
|
Fraction of wall time the decode thread was actively working. |
|
|
Fraction of wall time the detection thread was actively working. |
|
|
Fraction of wall time the embed+store thread was actively working. |
|
Stage throughput#
The stage_throughput block reports the per-stage processing rate, making it easy to spot which stage is the bottleneck.
Field |
Description |
Formula |
|---|---|---|
|
Frame decode rate (frames/s). |
|
|
Detection throughput (frames/s). |
|
|
Preprocessing throughput (items/s). |
|
|
Inference throughput (items/s). |
|
|
End-to-end embedding stage throughput (items/s). |
|
|
VDMS write throughput (items/s). |
|
|
Overall pipeline throughput (frames/s). Same value as |
|
Batch breakdown#
When batching is enabled, each entry in the batches array reports per-batch timing and counts. These entries make it easy to identify skewed batches (for example, ones with large detection times because of busy scenes).
Field |
Description |
|---|---|
|
Identifier of the stream this batch belongs to. |
|
Zero-based sequential identifier for the batch within its stream. |
|
Number of raw frames fed into this batch before detection. |
|
Frames and crops that passed the detection filter ( |
|
Time spent running object detection for this batch. |
|
Total embedding time for this batch (preprocessing + inference). |
|
Time spent on image preprocessing within the embedding step. |
|
Time spent on model inference within the embedding step. |
|
Time spent writing this batch’s embeddings to VDMS. |
|
Total end-to-end time for this batch (detect + embed + store). |
|
Number of embeddings successfully stored for this batch. |