# ViPPET Backend The ViPPET backend (the `vippet-app` service) is a Python application that exposes the REST API used by the UI, manages pipelines and models, and runs GStreamer / DL Streamer workloads on the underlying hardware. It is designed to keep request handling fast and non-blocking, while the actual media processing happens in a separate DL Streamer Pipeline Server 2.0 container. ## Core Stack ### Python 3.12 + FastAPI The service is built on FastAPI running under uvicorn. Route handlers are `async`, request and response bodies are described with Pydantic v2 models, and the OpenAPI schema is generated automatically. Swagger UI is available at `/docs` and ReDoc at `/redoc` (both proxied through the UI nginx, and reachable directly on port 7860). ### Pydantic v2 Schemas All API payloads are defined as Pydantic v2 models in `vippet/api/api_schemas.py`. The same schemas are reused for validation, OpenAPI generation, and as the source of the typed RTK Query client consumed by the UI. See [Backend contributing guide](../contributing/backend.md) for the regeneration workflow. ### OpenVINO, DL Streamer, GStreamer Inference workloads are executed by GStreamer pipelines that use DL Streamer elements (`gva*`) backed by OpenVINO™. Available hardware (CPU / GPU / NPU) is probed at startup through the OpenVINO™ Core in `vippet/device.py`, and exposed through the `/devices` endpoint so the UI can offer the right per-device pipeline variants. The CPU entry also carries the core type (P, E, LPE) of every online logical CPU, read from Linux sysfs by `vippet/cpu_topology.py`. ### Pipeline Execution on DL Streamer Pipeline Server 2.0 The backend does not run pipelines in its own process. Every pipeline run (validation, performance and density tests, benchmarks, and the conversion of videos to `.ts`) is sent over REST to DL Streamer Pipeline Server 2.0 (the `dlstreamer-pipeline-server` service) by `Dlsps2PipelineRunner` in `vippet/dlsps2_runner.py`. Each stream runs as its own server instance; the runner polls the per-stream FPS and judges a run by its slowest stream. Before a performance or density run, the runner validates stream 0 of each pipeline once, with its file, stream and metadata outputs swapped for `fakesink` so the check leaves nothing behind, and then starts every stream without the server's own pre-check. When the backend starts and when it stops, it stops every instance still active on the server and deletes the live-event folders, since it is the server's only client. Latency and VLM metrics come from the server's OpenTelemetry export to `metrics-manager`. Live pipeline events (proximity triggers, VLM answers) are written by the pipelines as JSON Lines files into the shared metadata folder and read back by `vippet/pipeline_events.py`. `vippet/pipeline_runner.py` holds the result types shared by both modules. The device list (`vippet/device.py`) is read from the server's `GET /devices`, so it shows the devices pipelines actually run on. The DL Streamer pipeline optimizer runs as a server job too: `vippet/dlsps2_optimizer.py` starts it with `POST /optimize`, polls it until it ends and cancels it on request, so the candidate pipelines also run on the server's runtime and devices. ### Docker Compose with Hardware Profiles The backend ships as a container and is composed with sibling services (`vippet-ui`, `dlstreamer-pipeline-server`, `mediamtx`, `model-download`, `metrics-manager`, `sensor-manager`, `audio-analyzer`, and `text-to-speech`) via Docker Compose. Hardware capabilities are selected through `COMPOSE_PROFILES` (`cpu`, `gpu`, `npu`, `gpu-wsl`), which is auto-detected by `setup_env.sh`. The WSL profile exposes `/dev/dxg` and read-only `/usr/lib/wsl` libraries instead of native DRM devices. ### Developer Tooling The backend uses `ruff` for linting and formatting, `pyright` for type checking, and `pytest` (with unit and functional suites) for tests. All targets are wired into the top-level `Makefile` (`make lint`, `make format`, `make test`, `make test-smoke`, `make test-full`). ## Runtime Architecture ### Layered Layout The Python package under `vippet/` is organized in clear layers: - `vippet/api/` - FastAPI app, routers, request / response schemas, and middleware. - `vippet/managers/` - business logic and in-memory state (singletons, thread-safe where needed). - `vippet/pipelines/` - built-in pipeline definitions (YAML) and the loader that turns them into runtime objects. - `vippet/graph.py`, `vippet/dlsps2_runner.py`, `vippet/pipeline_events.py`, `vippet/pipeline_runner.py` - pipeline graph model, pipeline execution on DL Streamer Pipeline Server 2.0, live pipeline events, and the run result types. - `vippet/device.py`, `vippet/cpu_topology.py`, `vippet/video_encoder.py`, `vippet/video_decoder.py`, `vippet/camera.py`, `vippet/images.py`, `vippet/videos.py` - hardware and media helpers used by managers and runners. ### Application Startup On startup the FastAPI `lifespan` hook (`vippet/api/main.py`) registers only the `health` router and immediately starts a background initialization thread. That thread sequentially brings up `VideosManager`, `ImagesManager`, `PipelineManager`, `PipelineTemplateManager`, and `ModelManager`, then registers the remaining routers. The `AppStateManager` exposes the current phase (`STARTING`, `INITIALIZING`, `READY`, `SHUTDOWN`) through the health endpoint, and the `InitializationMiddleware` rejects business calls until the app is `READY`. This pattern lets Docker health checks succeed early without serving partially initialized state. ### Request Lifecycle A typical request flows through these stages: 1. The UI calls `/api/v1/...` (nginx proxies API calls to the backend on port 7860). 1. FastAPI validates the body and path / query parameters against Pydantic models. 1. The route handler delegates to a manager. Managers hold state and own all business rules. 1. For long-running or media-bound work (optimization, validation, density / performance tests), the manager submits a job and returns a `job_id`. The actual execution happens in a worker thread that drives one or more runs on DL Streamer Pipeline Server 2.0 through `Dlsps2PipelineRunner`. 1. The UI polls the corresponding `/jobs/...` endpoint or subscribes to the SSE metrics stream for progress. ### Pipelines and Variants A pipeline is a logical workload (for example "License plate recognition") with one or more device-specific variants (`CPU`, `GPU`, `NPU`, `GPU_NPU`). Built-in pipelines are YAML files under `vippet/pipelines/`, auto-discovered by `PipelineLoader.list()` and registered as `read_only` (source `PREDEFINED`). User-defined pipelines are created at runtime through `POST /pipelines` (source `USER_CREATED`). The same `PipelineManager` owns both kinds. See the [New pipeline guide](../contributing/new-pipeline.md) for the YAML schema and REST payload. ### Pipeline Graph Model Each variant carries a full GStreamer description and a graph representation (`pipeline_graph`) plus a simplified view (`pipeline_graph_simple`) used by the visual editor. `vippet/graph.py` converts between the GStreamer description and the graph form, and applies the `SIMPLE_VIEW_VISIBLE_ELEMENTS` / `SIMPLE_VIEW_INVISIBLE_ELEMENTS` filtering rules that decide which elements appear in the simplified canvas. ### Automatic Graph Transformations `PipelineManager.build_pipeline_command()` does not run the variant graph as-is. For every stream it applies a small set of deterministic transformations defined in `vippet/graph.py`, so the same graph can be reused across different output modes without changes to the YAML or the saved user definition: - **Intermediate sink rewriting** (`prepare_intermediate_output_sinks`) — gives every non-fakesink terminal a unique, stream-scoped file location under the job output directory. - **Main output placeholder** (`prepare_main_output_placeholder`) — when `output_mode` is `file` or `live_stream`, the default `fakesink name=default_output_sink` is converted to an `OUTPUT_PLACEHOLDER` node. It is replaced later by the encoder + filesink (for `file`) or by the encoder + rtspclientsink (for `live_stream`) subpipeline built by `video_encoder`. - **Metadata file injection** (`inject_metadata_file_paths`) — when `metadata_mode=file`, every `gvametapublish` node gets a unique file path under the job metadata directory. - **Automatic `gvawatermark` stripping** (`strip_watermark_if_all_sinks_are_fake`) — removes every `gvawatermark` node when the only terminals left are fakesinks, i.e., there is no rendered video output that would consume the overlay. The watermark is preserved whenever an `OUTPUT_PLACEHOLDER` is present (`output_mode=file` or `live_stream`) or any non-fakesink terminal exists in the graph (for example a `splitmuxsink` in NVR-style pipelines that records the stream itself). This keeps the default `output_mode=disabled` measurement free of the overlay-rendering cost without affecting live view, file output, or pipelines that persist the video. - **Unique element names and source/sink identifiers** (`unify_all_element_names`, `apply_stream_identifiers`) — assigns deterministic, stream-unique names to every element so the latency tracer can correlate its rows back to a specific source/sink pair. ### Job and Metrics Flow Tests, optimization runs, and validation runs are tracked as jobs. Each job has a status (`PENDING`, `RUNNING`, `COMPLETED`, `FAILED`, `CANCELLED`), a structured summary, and per-stream metadata. Live runtime metrics (CPU, GPU, NPU usage, power, memory, FPS) are collected by the sibling `metrics-manager` service from Telegraf and qmassa, and streamed to the UI as Server-Sent Events through the nginx proxy at `/metrics/stream`. Per-frame inference metadata produced by DL Streamer is captured by `MetadataManager`, which tails the metadata files and streams records to the UI through SSE. ## Managers The `managers/` package holds the business-logic layer. Each manager is responsible for one domain and is the only component allowed to mutate state for that domain. Most managers are thread-safe singletons. ### AppStateManager Tracks the lifecycle state of the application (`STARTING`, `INITIALIZING`, `READY`, `SHUTDOWN`) and an optional human-readable message. Used by the health endpoint and by `InitializationMiddleware` to gate traffic. Key methods: `set_status()`, `status()`, `is_ready()`, `is_healthy()`. ### PipelineManager The central registry of pipelines and variants. Loads predefined pipelines from YAML on startup, accepts user-created pipelines through the API, and builds the runtime command lines used by the pipeline runner. Key methods: `add_pipeline()`, `get_pipelines()`, `get_pipeline_by_id()`, `update_pipeline()`, `delete_pipeline_by_id()`, `add_variant()`, `update_variant()`, `delete_variant()`, `load_predefined_pipelines()`, `build_pipeline_command()`, and the `validate_and_convert_advanced_to_simple()` / `validate_and_convert_simple_to_advanced()` graph converters. ### PipelineTemplateManager Read-only registry of reusable pipeline templates that the UI offers as starting points when creating a new user-defined pipeline. Key methods: `get_templates()`, `get_template_by_id()`. ### ModelManager Owns the model catalog (`vippet/models/*.yaml`, seeded into the DB at startup) and tracks install state per model. Triggers downloads through the `model-download` microservice, tracks download jobs, supports user uploads of custom models, and computes the `used_by_pipelines` field shown in the UI. Key methods: `list_models()`, `start_download()`, `upload_model()`, `get_job()`, `get_all_jobs()`. ### CameraManager Provides media input sources discovered by the `sensor-manager` microservice: local USB cameras under `/dev/video*` and ONVIF network cameras. Caches the camera list for pipeline graph lookups (device path or RTSP URL to capture format, encoding and credentials) and keeps ONVIF credentials entered by the user in memory only. Key methods: `discover_all_cameras()`, `get_camera_by_id()`, `load_camera_profiles()`, `get_usb_camera_details_by_device_path()`, `get_network_camera_details_by_rtsp_url()`, `get_encoding_for_rtsp_url()`. Custom RTSP URLs that do not belong to a discovered camera are handled by `rtsp_utils.py`: `validate_rtsp_url()` rejects URLs that could alter the generated pipeline description (whitespace, `!`, quotes, `,` and similar) or that embed credentials, and `probe_rtsp_codec()` detects the stream codec with `gst-discoverer-1.0` (timeout `RTSP_PROBE_TIMEOUT_S`, results cached per URL) so that `Graph` can select a device-specific decoder instead of `decodebin3`. ### TestsManager Runs performance and density benchmarks. Performance tests run a single pipeline variant under defined conditions; density tests use the algorithm in `benchmark.py` to find the maximum number of concurrent streams the hardware can sustain. Density supports two modes selected automatically from the request shape: **classic mode** (search variable = total stream count, distributed across pipelines by `stream_rate`) and **mixed mode** (exactly two pipelines; one pinned to a fixed `streams` value, the other incremented by the search variable). Each invocation returns a `job_id` that the UI can poll, summarize, or cancel. Key methods: `test_performance()`, `test_density()`, `get_job_status()`, `get_job_summary()`, `get_job_statuses_by_type()`, `stop_job()`. ### OptimizationManager Coordinates model optimization runs (preprocessing + optimization), driven by the `OptimizationRunner` helper. Designed for long-running, cancellable jobs and exposes status / summary endpoints used by the UI to render progress. Key methods: `run_optimization()`, `get_job_status()`, `get_all_job_statuses()`, `get_job_summary()`. ### ValidationManager Runs accuracy and correctness checks against pipeline variants (for example comparing inference output against a reference) and tracks each run as a job. Key methods: `run_validation()`, `get_job_status()`, `get_all_job_statuses()`, `get_job_summary()`. ### MetadataManager Tails the per-job DL Streamer metadata files written during pipeline runs and exposes them to the UI both as paged snapshots and as live Server-Sent Events streams. Key methods: `register_job()`, `get_snapshot()`, `resolve_file_index()`, `stream_events()`, `stop_tailing()`. ## Sibling Services and Integrations The backend does not run alone. It cooperates with a small set of sibling containers managed by Docker Compose: - **`dlstreamer-pipeline-server`** (DL Streamer Pipeline Server 2.0, built locally from `microservices/dlstreamer-pipeline-server/dlsps2` unless `DLSPS2_IMAGE` names a published image) - runs every pipeline the backend builds, reached at `DLSPS2_BASE_URL` (default `http://dlstreamer-pipeline-server:8080`). It gets the same devices as the backend and loads the custom elements of `vippet/gst_plugins/`, which the one-shot **`vippet-plugins`** service copies from the backend image into the `vippet-gst-plugins` Docker volume. - **`vippet-ui`** - serves the React app and proxies `/api/v1/*`, `/metrics/*`, and `/docs` to the backend on port 7860. - **`mediamtx`** - RTSP server used for live preview streams. The backend publishes live encoded output to `rtsp://mediamtx:8554/` (configurable through `LIVE_STREAM_SERVER_HOST` / `LIVE_STREAM_SERVER_PORT`). - **`model-download`** - dedicated microservice that performs the actual model downloads. `ModelManager` calls into it and polls job status; the backend exposes a stable `/models` API on top. - **`metrics-manager`** - collects hardware telemetry via Telegraf and qmassa and pushes it to clients over SSE. The UI subscribes to `/metrics/stream`, which nginx routes to this service. - **`sensor-manager`** (built locally from `microservices/sensor-manager`) - discovers USB cameras and ONVIF network cameras and loads their ONVIF media profiles. It runs with host networking (required by WS-Discovery multicast); `CameraManager` reaches it at `SENSOR_MANAGER_URL` (default `http://host.docker.internal:8090`) and exposes the stable `/cameras` API on top. ## Summary The ViPPET backend keeps a clear separation between a thin async FastAPI surface and a manager-centric business-logic layer, while pushing all heavy media work to DL Streamer Pipeline Server 2.0. Pipelines, models, cameras, jobs, and metrics each have a dedicated manager with a well-defined API, and the OpenAPI schema (browsable at `/docs`) is the contract that ties everything to the typed UI client. For day-to-day work on the backend, start with the [Backend contributing guide](../contributing/backend.md); to add a new workload, see [Adding a new pipeline](../contributing/new-pipeline.md).