ViPPET Backend#
The ViPPET backend (the vippet-app service) is a Python application that exposes the REST API used by the
UI, manages pipelines and models, and runs GStreamer / DL Streamer workloads on the underlying hardware. It is
designed to keep request handling fast and non-blocking, while the actual media processing happens in a
separate DL Streamer Pipeline Server 2.0 container.
Core Stack#
Python 3.12 + FastAPI#
The service is built on FastAPI running under uvicorn. Route handlers are async, request and response bodies
are described with Pydantic v2 models, and the OpenAPI schema is generated automatically. Swagger UI is
available at /docs and ReDoc at /redoc (both proxied through the UI nginx, and reachable directly on port
7860).
Pydantic v2 Schemas#
All API payloads are defined as Pydantic v2 models in vippet/api/api_schemas.py. The same schemas are reused
for validation, OpenAPI generation, and as the source of the typed RTK Query client consumed by the UI. See
Backend contributing guide for the regeneration workflow.
OpenVINO, DL Streamer, GStreamer#
Inference workloads are executed by GStreamer pipelines that use DL Streamer elements (gva*) backed by
OpenVINO™. Available hardware (CPU / GPU / NPU) is probed at startup through the OpenVINO™ Core in
vippet/device.py, and exposed through the /devices endpoint so the UI can offer the right per-device
pipeline variants. The CPU entry also carries the core type (P, E, LPE) of every online logical CPU, read
from Linux sysfs by vippet/cpu_topology.py.
Pipeline Execution on DL Streamer Pipeline Server 2.0#
The backend does not run pipelines in its own process. Every pipeline run (validation, performance and density
tests, benchmarks, and the conversion of videos to .ts) is sent over REST to DL Streamer Pipeline Server 2.0
(the dlstreamer-pipeline-server service) by Dlsps2PipelineRunner in vippet/dlsps2_runner.py. Each stream
runs as its own server instance; the runner polls the per-stream FPS and judges a run by its slowest stream.
Before a performance or density run, the runner validates stream 0 of each pipeline once, with its file, stream
and metadata outputs swapped for fakesink so the check leaves nothing behind, and then starts every stream
without the server’s own pre-check. When the backend starts and when it stops, it stops every instance still
active on the server and deletes the live-event folders, since it is the server’s only client.
Latency and VLM metrics come from the server’s OpenTelemetry export to metrics-manager. Live pipeline events
(proximity triggers, VLM answers) are written by the pipelines as JSON Lines files into the shared metadata
folder and read back by vippet/pipeline_events.py. vippet/pipeline_runner.py holds the result types shared
by both modules.
The device list (vippet/device.py) is read from the server’s GET /devices, so it shows the devices pipelines
actually run on. The DL Streamer pipeline optimizer runs as a server job too: vippet/dlsps2_optimizer.py starts it
with POST /optimize, polls it until it ends and cancels it on request, so the candidate pipelines also run on the
server’s runtime and devices.
Docker Compose with Hardware Profiles#
The backend ships as a container and is composed with sibling services (vippet-ui,
dlstreamer-pipeline-server, mediamtx, model-download, metrics-manager, sensor-manager,
audio-analyzer, and text-to-speech) via Docker Compose. Hardware capabilities are selected through
COMPOSE_PROFILES (cpu, gpu, npu, gpu-wsl), which is auto-detected by setup_env.sh.
The WSL profile exposes /dev/dxg and read-only /usr/lib/wsl libraries instead of native DRM devices.
Developer Tooling#
The backend uses ruff for linting and formatting, pyright for type checking, and pytest (with unit and
functional suites) for tests. All targets are wired into the top-level Makefile (make lint, make format,
make test, make test-smoke, make test-full).
Runtime Architecture#
Layered Layout#
The Python package under vippet/ is organized in clear layers:
vippet/api/- FastAPI app, routers, request / response schemas, and middleware.vippet/managers/- business logic and in-memory state (singletons, thread-safe where needed).vippet/pipelines/- built-in pipeline definitions (YAML) and the loader that turns them into runtime objects.vippet/graph.py,vippet/dlsps2_runner.py,vippet/pipeline_events.py,vippet/pipeline_runner.py- pipeline graph model, pipeline execution on DL Streamer Pipeline Server 2.0, live pipeline events, and the run result types.vippet/device.py,vippet/cpu_topology.py,vippet/video_encoder.py,vippet/video_decoder.py,vippet/camera.py,vippet/images.py,vippet/videos.py- hardware and media helpers used by managers and runners.
Application Startup#
On startup the FastAPI lifespan hook (vippet/api/main.py) registers only the health router and
immediately starts a background initialization thread. That thread sequentially brings up VideosManager,
ImagesManager, PipelineManager, PipelineTemplateManager, and ModelManager, then registers the
remaining routers. The AppStateManager exposes the current phase (STARTING, INITIALIZING, READY,
SHUTDOWN) through the health endpoint, and the InitializationMiddleware rejects business calls until the
app is READY. This pattern lets Docker health checks succeed early without serving partially initialized
state.
Request Lifecycle#
A typical request flows through these stages:
The UI calls
/api/v1/...(nginx proxies API calls to the backend on port 7860).FastAPI validates the body and path / query parameters against Pydantic models.
The route handler delegates to a manager. Managers hold state and own all business rules.
For long-running or media-bound work (optimization, validation, density / performance tests), the manager submits a job and returns a
job_id. The actual execution happens in a worker thread that drives one or more runs on DL Streamer Pipeline Server 2.0 throughDlsps2PipelineRunner.The UI polls the corresponding
/jobs/...endpoint or subscribes to the SSE metrics stream for progress.
Pipelines and Variants#
A pipeline is a logical workload (for example “License plate recognition”) with one or more device-specific
variants (CPU, GPU, NPU, GPU_NPU). Built-in pipelines are YAML files under vippet/pipelines/,
auto-discovered by PipelineLoader.list() and registered as read_only (source PREDEFINED). User-defined
pipelines are created at runtime through POST /pipelines (source USER_CREATED). The same PipelineManager
owns both kinds. See the New pipeline guide for the YAML schema and REST
payload.
Pipeline Graph Model#
Each variant carries a full GStreamer description and a graph representation (pipeline_graph) plus a
simplified view (pipeline_graph_simple) used by the visual editor. vippet/graph.py converts between the
GStreamer description and the graph form, and applies the SIMPLE_VIEW_VISIBLE_ELEMENTS /
SIMPLE_VIEW_INVISIBLE_ELEMENTS filtering rules that decide which elements appear in the simplified canvas.
Automatic Graph Transformations#
PipelineManager.build_pipeline_command() does not run the variant graph as-is. For every stream it
applies a small set of deterministic transformations defined in vippet/graph.py, so the same graph can
be reused across different output modes without changes to the YAML or the saved user definition:
Intermediate sink rewriting (
prepare_intermediate_output_sinks) — gives every non-fakesink terminal a unique, stream-scoped file location under the job output directory.Main output placeholder (
prepare_main_output_placeholder) — whenoutput_modeisfileorlive_stream, the defaultfakesink name=default_output_sinkis converted to anOUTPUT_PLACEHOLDERnode. It is replaced later by the encoder + filesink (forfile) or by the encoder + rtspclientsink (forlive_stream) subpipeline built byvideo_encoder.Metadata file injection (
inject_metadata_file_paths) — whenmetadata_mode=file, everygvametapublishnode gets a unique file path under the job metadata directory.Automatic
gvawatermarkstripping (strip_watermark_if_all_sinks_are_fake) — removes everygvawatermarknode when the only terminals left are fakesinks, i.e., there is no rendered video output that would consume the overlay. The watermark is preserved whenever anOUTPUT_PLACEHOLDERis present (output_mode=fileorlive_stream) or any non-fakesink terminal exists in the graph (for example asplitmuxsinkin NVR-style pipelines that records the stream itself). This keeps the defaultoutput_mode=disabledmeasurement free of the overlay-rendering cost without affecting live view, file output, or pipelines that persist the video.Unique element names and source/sink identifiers (
unify_all_element_names,apply_stream_identifiers) — assigns deterministic, stream-unique names to every element so the latency tracer can correlate its rows back to a specific source/sink pair.
Job and Metrics Flow#
Tests, optimization runs, and validation runs are tracked as jobs. Each job has a status (PENDING,
RUNNING, COMPLETED, FAILED, CANCELLED), a structured summary, and per-stream metadata. Live runtime
metrics (CPU, GPU, NPU usage, power, memory, FPS) are collected by the sibling metrics-manager service from
Telegraf and qmassa, and streamed to the UI as Server-Sent Events through the nginx proxy at
/metrics/stream. Per-frame inference metadata produced by DL Streamer is captured by MetadataManager, which
tails the metadata files and streams records to the UI through SSE.
Managers#
The managers/ package holds the business-logic layer. Each manager is responsible for one domain and is the
only component allowed to mutate state for that domain. Most managers are thread-safe singletons.
AppStateManager#
Tracks the lifecycle state of the application (STARTING, INITIALIZING, READY, SHUTDOWN) and an
optional human-readable message. Used by the health endpoint and by InitializationMiddleware to gate
traffic. Key methods: set_status(), status(), is_ready(), is_healthy().
PipelineManager#
The central registry of pipelines and variants. Loads predefined pipelines from YAML on startup, accepts
user-created pipelines through the API, and builds the runtime command lines used by the pipeline runner. Key
methods: add_pipeline(), get_pipelines(), get_pipeline_by_id(), update_pipeline(),
delete_pipeline_by_id(), add_variant(), update_variant(), delete_variant(),
load_predefined_pipelines(), build_pipeline_command(), and the validate_and_convert_advanced_to_simple()
/ validate_and_convert_simple_to_advanced() graph converters.
PipelineTemplateManager#
Read-only registry of reusable pipeline templates that the UI offers as starting points when creating a new
user-defined pipeline. Key methods: get_templates(), get_template_by_id().
ModelManager#
Owns the model catalog (vippet/models/*.yaml, seeded into the DB at startup) and tracks install state per model.
Triggers downloads through the model-download microservice, tracks download jobs, supports user uploads of
custom models, and computes the used_by_pipelines field shown in the UI. Key methods: list_models(),
start_download(), upload_model(), get_job(), get_all_jobs().
CameraManager#
Provides media input sources discovered by the sensor-manager microservice: local USB cameras under
/dev/video* and ONVIF network cameras. Caches the camera list for pipeline graph lookups (device path or
RTSP URL to capture format, encoding and credentials) and keeps ONVIF credentials entered by the user in
memory only. Key methods: discover_all_cameras(), get_camera_by_id(), load_camera_profiles(),
get_usb_camera_details_by_device_path(), get_network_camera_details_by_rtsp_url(),
get_encoding_for_rtsp_url().
Custom RTSP URLs that do not belong to a discovered camera are handled by rtsp_utils.py:
validate_rtsp_url() rejects URLs that could alter the generated pipeline description (whitespace, !,
quotes, , and similar) or that embed credentials, and probe_rtsp_codec() detects the stream codec with
gst-discoverer-1.0 (timeout RTSP_PROBE_TIMEOUT_S, results cached per URL) so that Graph can select a
device-specific decoder instead of decodebin3.
TestsManager#
Runs performance and density benchmarks. Performance tests run a single pipeline variant under defined
conditions; density tests use the algorithm in benchmark.py to find the maximum number of concurrent streams
the hardware can sustain. Density supports two modes selected automatically from the request shape:
classic mode (search variable = total stream count, distributed across pipelines by stream_rate) and
mixed mode (exactly two pipelines; one pinned to a fixed streams value, the other incremented by the
search variable). Each invocation returns a job_id that the UI can poll, summarize, or cancel. Key
methods: test_performance(), test_density(), get_job_status(), get_job_summary(),
get_job_statuses_by_type(), stop_job().
OptimizationManager#
Coordinates model optimization runs (preprocessing + optimization), driven by the OptimizationRunner helper.
Designed for long-running, cancellable jobs and exposes status / summary endpoints used by the UI to render
progress. Key methods: run_optimization(), get_job_status(), get_all_job_statuses(),
get_job_summary().
ValidationManager#
Runs accuracy and correctness checks against pipeline variants (for example comparing inference output against
a reference) and tracks each run as a job. Key methods: run_validation(), get_job_status(),
get_all_job_statuses(), get_job_summary().
MetadataManager#
Tails the per-job DL Streamer metadata files written during pipeline runs and exposes them to the UI both as
paged snapshots and as live Server-Sent Events streams. Key methods: register_job(), get_snapshot(),
resolve_file_index(), stream_events(), stop_tailing().
Sibling Services and Integrations#
The backend does not run alone. It cooperates with a small set of sibling containers managed by Docker Compose:
dlstreamer-pipeline-server(DL Streamer Pipeline Server 2.0, built locally frommicroservices/dlstreamer-pipeline-server/dlsps2unlessDLSPS2_IMAGEnames a published image) - runs every pipeline the backend builds, reached atDLSPS2_BASE_URL(defaulthttp://dlstreamer-pipeline-server:8080). It gets the same devices as the backend and loads the custom elements ofvippet/gst_plugins/, which the one-shotvippet-pluginsservice copies from the backend image into thevippet-gst-pluginsDocker volume.vippet-ui- serves the React app and proxies/api/v1/*,/metrics/*, and/docsto the backend on port 7860.mediamtx- RTSP server used for live preview streams. The backend publishes live encoded output tortsp://mediamtx:8554/<stream-name>(configurable throughLIVE_STREAM_SERVER_HOST/LIVE_STREAM_SERVER_PORT).model-download- dedicated microservice that performs the actual model downloads.ModelManagercalls into it and polls job status; the backend exposes a stable/modelsAPI on top.metrics-manager- collects hardware telemetry via Telegraf and qmassa and pushes it to clients over SSE. The UI subscribes to/metrics/stream, which nginx routes to this service.sensor-manager(built locally frommicroservices/sensor-manager) - discovers USB cameras and ONVIF network cameras and loads their ONVIF media profiles. It runs with host networking (required by WS-Discovery multicast);CameraManagerreaches it atSENSOR_MANAGER_URL(defaulthttp://host.docker.internal:8090) and exposes the stable/camerasAPI on top.
Summary#
The ViPPET backend keeps a clear separation between a thin async FastAPI surface and a manager-centric
business-logic layer, while pushing all heavy media work to DL Streamer Pipeline Server 2.0. Pipelines,
models, cameras, jobs, and metrics each have a dedicated manager with a well-defined API, and the OpenAPI
schema (browsable at /docs) is the contract that ties everything to the typed UI client. For day-to-day work
on the backend, start with the Backend contributing guide; to add a new
workload, see Adding a new pipeline.