How It Works#

This section provides a high-level view of how the application processes audio and video input through two parallel pipelines integrated with a modular backend architecture.

High-Level System Diagram

Inputs#

You can upload audio recordings and video files, or provide RTSP video streams through the Web-based UI layer, which supports:

  • Audio and video upload

  • Viewing transcription, summaries, mind maps, and classroom statistics

  • Monitoring video analytics streams with real-time overlays

  • Localization options (English/Chinese)

The uploaded media is passed to the Backend API, which acts as the gateway to the backend service layer and provides similar capabilities.

Audio Pipeline#

The audio pipeline handles speech-to-text conversion and content summarization:

  • Audio Pre-processing Cleans and formats audio data using FFmpeg, chunking input into segments for optimal processing.

  • ASR Component (Automatic Speech Recognition) Converts audio into text through the Audio Analyzer microservice, which provides transcription and optional speaker diarization. The provider is set by models.asr.provider in config.yaml. On Linux, Audio Analyzer requests its models from the shared model-downloader container on first start and reuses them from the model-cache volume afterwards:

    Provider

    Models

    Downloaded from

    OpenVINO™ (default)

    Whisper (e.g. whisper-base)

    Hugging Face (openai/whisper-*) via model-downloader, exported to OpenVINO IR

    FunASR

    Paraformer (paraformer-zh), for Mandarin

    ModelScope via model-downloader. On Windows, model_hub: huggingface moves only the VAD and CAM++ models to Hugging Face

    OpenAI

    Whisper (e.g. whisper-base)

    OpenAI’s public openai-whisper checkpoints (fetched directly)

  • Speaker Diarization Identifies and separates individual speakers using Pyannote Audio models. This can be enabled or disabled by modifying config.yaml.

  • Summarizer Component Generates concise summaries of transcribed text using the OpenVINO™ LLM provider.

  • Content Segmentation The LLM segments the transcript into 15–25 topic-based sections, each of which is encoded and indexed into the vector store. Users can then search lecture content by natural-language queries, retrieving the most relevant topic segments.

Video Analytics Pipeline#

The Video Analytics (VA) pipeline performs real-time video analysis using Deep Learning Streamer (DL Streamer) and OpenVINO™, processing multiple concurrent video streams to extract classroom engagement data.

Pipeline Architecture#

The VA pipeline runs three independent processing streams simultaneously:

  • Front Video Pipeline — Student-facing camera stream for detailed person tracking, pose estimation, posture classification, hand-raise recognition, and person re-identification

  • Back Video Pipeline — Rear-view camera stream for broader classroom monitoring and basic pose analysis

  • Content Pipeline — Interactive flat panel display (IFPD) or board capture, processed at 1 FPS for content frame analysis

Each pipeline is built as a DL Streamer processing graph with the following stages:

  1. Video Decode and Preprocessing — Input from RTSP stream or video file is decoded and preprocessed with hardware acceleration (D3D11)

  2. Person Detection and Pose Estimation — YOLO models (YOLOv8m-pose for front, YOLOv8s-pose for back) detect persons and estimate 17-keypoint skeletons per frame

  3. Posture Detection — A custom DL Streamer element analyzes keypoint geometry to classify posture (sit/stand) and detect hand-raises

  4. Re-identification and Classification:

    • Person-ReID-retail-0288 re-identifies standing students across frames (front pipeline)

    • ResNet-18 classifies whole frames of the content pipeline

    • On Linux, the front and back pipelines also run per-person ResNet-18 classification, and the front pipeline MobileNet-V2 classification; the Windows application does not run these branches

  5. Output — Annotated video is streamed via RTSP through a MediaMTX media server; per-frame metadata is written as JSON for statistics aggregation

Classroom Statistics#

The VA pipeline aggregates per-frame metadata into classroom engagement statistics:

  • Student count — Number of students detected by the front pipeline. On Windows, each one-minute window takes the median of its per-frame counts, which smooths out frames where a student is briefly missed or detected twice, and the class value is the median over all windows, so the minutes when students are arriving or leaving do not skew it. The window length and percentile are set by count_bucket_sec and count_percentile under va_pipeline.pose_statistics in config.yaml. On Linux, the count is the average over the most recent 150 frames (VA_POSE_STUDENT_COUNT_WINDOW_FRAMES).

  • Stand-up events — Per-student stand-up detection with noise filtering

  • Hand-raise events — Per-student hand-raise tracking with configurable confirmation thresholds

  • Per-student tracking — Re-identified students are tracked across frames with unique IDs

Streaming and Distribution#

A Media Server (MediaMTX) receives processed video from all three pipelines and provides:

  • RTSP streaming for real-time playback

  • HLS/WebRTC streaming for browser-based viewing

Board Content OCR#

Extracts text from the IFPD or whiteboard captured by the content pipeline, enabling post-class review of all written or projected content.

  • Frame Extraction — FFmpeg (Intel QSV hardware decode) samples JPEG frames at a configurable rate (default: 1 fps every 3 seconds).

  • Deduplication — Frames with fewer than 0.1% of pixels changed by more than 25 intensity units are skipped.

  • OCR and Confidence Gating — PaddleOCR extracts text; frames with mean confidence below threshold are dropped.

  • Post-processing — Near-duplicate consecutive records are merged to reduce downstream LLM token usage.

  • Output — Results are written as JSON-lines to board_ocr.txt ({frame, source_frame, timestamp, text}).

  • Summary integration — When Board OCR has captured text for a session, the audio summarization step folds it into the AI-generated class summary as an extra “Board / IFPD Content” section, alongside the sections derived from the transcript. If no board text is available, the summary is unchanged.

Content Search Pipeline#

The Content Search pipeline provides multimodal ingestion, semantic indexing, and retrieval across videos, documents, and images.

Content Search Architecture

Content Search Software Stack

Ingestion#

  • Video: Split into time-based chunks (default 30s with 4s overlap), and each chunk’s sampled frames are summarized by a Vision Language Model (Qwen2.5-VL via OpenVINO™). Summaries are indexed as text embeddings; frames are indexed as visual embeddings.

  • Document: Full-text extraction via unstructured with image OCR via PaddleOCR service. Text is split using semantic chunking (embedding-based boundary detection) or fixed-size chunking, then embedded with BGE (bge-small-en-v1.5).

  • Image: Embedded directly via CLIP (xlm-roberta-base-ViT-B-32) for visual similarity search.

All embeddings are stored in ChromaDB across two collections: a visual collection (CLIP embeddings for images/video frames) and a textual collection (BGE embeddings for document chunks/video summaries).

Question and Answer (RAG)#

The Q&A endpoint retrieves top-k relevant chunks, assembles them as context within a token budget, and sends them together with the user’s question to the VLM to generate a grounded answer with source references.

Microservices#

Service

Port

Role

Content Search API

9011

Orchestrator and public API

File Ingest & Retrieve

9990

Embedding, indexing, and retrieval

Video Preprocess

8001

Video chunking and VLM summarization

Smart Classroom backend

8000

Vision-language model inference (shared with the main app)

ChromaDB

9090

Vector database

For API details, see the Content Search Dev Guide.

Metrics Collector#

Monitors and collects:

  • xPU utilization for hardware performance

  • LLM metrics for summarization efficiency

Outputs#

  • Transcriptions, summaries, mind maps, and topic segments can be accessed from the Web-based UI and file system. The path for the file system is /<project-location>/<your-project-name>/. For example, /storage/chapter-10/

  • Semantic topic search results are returned via the API, with similarity scores and time-range references into the original recording.

  • Classroom statistics (student count, stand-up events, hand-raise events) are generated from the video analytics pipeline and displayed in the UI.

  • Video streams with real-time detection overlays are available via RTSP and HLS/WebRTC.

  • Performance metrics (e.g., utilization, model efficiency) are displayed for monitoring.

  • Localization ensures outputs are available in multiple languages (English/Chinese).

Learn More#